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THE  USE  OF  PILOT  RATING  IN  THE  EVALUATION  OF 


AIRCRAFT  HANDLING  QUALITIES 
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Ames  Research  Center 
and 

Robert  P.  Harper,  Jr. 
Cornell  Aeronautical  Laboratory 


SUMMARY 


Pilot  rating  scales  and  their  use  in  assessing  aircraft  handling 
qualities  are  reviewed  historically,  and  objections  that  have  been  raised  to 
limitations  of  earlier  scales  are  considered  in  the  development  of  a  revised 
scale.  Terminology  used  in  the  evaluation  of  handling  qualities  is  reviewed 
and  new  definitions  are  proposed  to  improve  communication  and  international 
understanding.  Of  particular  significance  is  the  new  definition  of  handling 
qualities,  which  emphasizes  the  importance  of  factors  that  influence  the  selec¬ 
tion  of  a  rating  other  than  stability  and  control  characteristics . 

The  experimental  use  of  pilot  rating  is  discussed  in  detail,  with  special 
attention  devoted  to  (1)  clarifying  the  difference  between  mission  and  task, 

(2)  identifying  what  the  rating  applies  to,  (3)  considering  the  pilot's 
assessment  criteria,  and  (4)  defining  the  simulation  situation.  The  inqjortant 
elements  of  the  report  are  then  summarized  in  a  suggested  "Briefing  Guide," 
designed  for  guidance  in  planning  and  executing  handling  qualities 
experiments . 

INTRODUCTION 


The  widespread  application  of  pilot  rating  scales  in  the  evaluation  of 
aircraft  handling  qualities  has  confirmed  their  basic  utility,  but  has,  at  the 
same  time,  exposed  some  weaknesses  of  the  scales  as  originally  proposed.  It 
was  therefore  considered  desirable  to  re-examine  existing  rating  scales  with 
the  purpose  of  developing  a  single  in5)roved  scale  and  of  clarifying  its  use  in 
the  evaluation  of  handling  qualities.  In  response  to  an  invitation  from  the 
Flight  Mechanics  Panel,  AGARD,  a  paper  entitled,  "A  Revised  Pilot  Rating  Scale 
for  the  Evaluation  of  Handling  Qualities,"  was  prepared  and  presented  at  the 
September  1966  meeting  (ref.  1).  A  longer  version  of  this  paper  (ref.  2)  con¬ 
tains  some  additional  explanatory  discussion  of  the  use  of  pilot  rating  scales. 
Additional  constructive  criticism  was  then  based  on  experience  gained  with 
this  revised  scale. 


In  general,  the  revised  scale  was  preferred  over  earlier  scales,  but 
constructive  criticism  from  many  research  and  development  groups  also 
indicated  the  need  for  additional  changes  or  clarifying  discussion.  Diffi¬ 
culties,  for  example,  were  experienced  with  the  semantics  in  that  certain 
words  had  rather  different  connotations  in  the  United  States,  England,  and  in 
France.  The  purpose  of  this  report,  then,  is  to  clarify  and  modify,  as  appro¬ 
priate,  the  material  presented  in  references  1  and  2.  One  of  the  first  objec¬ 
tives  of  the  present  report  is  to  define  precisely  the  basic  terminology  and 
explain  the  new  features  in  the  scale.  The  report  goes  on  to  discuss  the  more 
important  factors  that  are  considered  by  the  pilot  in  the  selection  of  a  rat¬ 
ing  or  that  will  otherwise  influence  the  rating.  Throughout  the  discussion, 
attention  is  directed  to  the  questions i  (1)  what  is  the  pilot  being  asked  to 
rate?,  and  (2)  how  will  the  e3q)erimental  results  be  used?  The  answers  to  both 
questions  have  important  bearing  on  the  interpretation  of  evaluations  made  by 
pilots  with  different  backgrounds,  e:q)erience,  and  points  of  view.  The  final 
section  of  the  discussion  is  devoted  to  a  review  of  certain  other  consider¬ 
ations  that  are  helpful  in  the  design  of  handling  qualities  e3q)eriments  and  to 
the  use  of  pilot  rating.  The  important  elements  of  the  report  are  summarized 
in  a  condensed  "Briefing  Guide"  for  use  in  planning  and  executing  handling 
qualities  experiments . 


DISCUSSION  OF  HANDLING  QUALITIES 


Clarification  of  Terms 

For  a  pilot  rating  scale  to  be  universally  acceptable  and  consistently 
applied  in  the  evaluation  of  handling  qualities,  the  terminology  must  be 
easily  understood  by  all  persons  working  in  the  field.  Those  terms  requiring 
specific  attention  are  defined  in  appendix  A,  and  several  definitions  are  sug¬ 
gested  that  may  help  clarify  and  standardize  the  terminology.  Some  of  the 
terms  suggested  in  reference  3  have  been  adopted  in  this  report.  Those  most 
significant  to  a  discussion  of  handling  qualities  are  examined  in  considerable 
detail  in  the  following  paragraphs.  Others  are  discussed  as  the  need  arises. 

Handling  qualities.-  The  term  "Handling  Qualities"  requires  a  clear 
definition  in  order  to  enphasize  that  it  includes  more  than  just  stability  and 
control  characteristics.  Other  factors  that  influence  the  handling  qualities 
are  the  cockpit  interface  (e.g.,  displays,  controls),  the  aircraft  environ¬ 
ment  (e.g.,  weather  conditions,  visibility,  turbulence)  and  stress,  the 
effects  of  which  cannot  readily  be  segregated.  Thus  in  most  tests,  handling 
qualities  are  really  being  evaluated  in  the  aggregate. 

In  appendix  A,  "Handling  Qualities"  is  defined  as  "those  qualities  or 
characteristics  of  an  aircraft  that  govern  the  ease  and  precision  with  which 
a  pilot  is  able  to  perform  the  tasks  required  in  support  of  an  aircraft  role." 
The  generally  accepted  meaning  of  "Flying  Qualities"  is  similar  to  this  defi¬ 
nition  of  "Handling  Qualities,"  so  only  the  latter  term  is  used  in  this 
report . 
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Figure  1  illustrates  the  factors  besides  stability  and  control  that 
influence  handling  qualities.  Here  the  primary  elements  of  the  pilot  control 
loop  are  arranged  to  illustrate  their  relationship  to  the  operation  of  the 
pilot-vehicle  combination.  In  addition  to  the  pilot,  the  task,  and  the 
stability  and  control  characteristics,  factors  shown  as  influencing  closure  of 
the  pilot  control  loops  are  the  cockpit  interface,  the  aircraft  environment, 
and  the  pilot's  stress. 
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Figure  1.-  Elements  of  control  loop  that  influence  handling  qualities. 


Mission.-  The  term  "mission"  has  been  used  in  the  United  States  rather 
loosely,  and  may  actually  have  several  meanings,  depending  on  how  it  is  used. 
"Mission"  has  been  used  to  identify,  in  a  general  sense,  the  purpose  or 
objective  for  which  an  aircraft  is  built.  It  has  also  been  used  to  designate 
a  complete  flight  or  sortie  or  even  an  undefined  part  of  the  flight.  By 
inference,  this  undefined  part  is  usually  the  special  flight  phase  during 
which  the  primary  assignment  is  carried  out. 

To  avoid  this  ambiguity,  the  terms  "role,"  "flight"  or  "sortie,"  "flight 
phase,"  and  "flight  subphase"  are  defined  for  use  in  place  of  "mission"  (or 
mission  element) .  The  continued  use  of  "mission"  in  relation  to  handling 
qualities,  however,  makes  it  worthwhile  to  have  an  acceptable  definition. 

One  that  has  been  suggested  (ref.  4)  is  "the  composite  of  pilot-vehicle  func¬ 
tions  that  must  be  performed  to  fulfill  operational  requirements."  In  the 
present  report,  "mission"  is  a  general  term  used  to  convey  this  concept  of 
'"operational  requirements,"  that  is,  the  objectives  or  delineation  of  what  it 
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is  that  the  pilot-vehicle  combination  must  be  able  to  accomplish.  ^  a  rule, 
the  mission  objectives  (required  operations)  are  cited  in  more  detail  as  the 
flight  segment  (flight  phase  or  subphase)  of  interest  becomes  more  specific. 

The  distinct  differences  intended  between  '•mission"  and  "task"  will  be 
clarified  in  the  following  paragraphs. 

Task.-  The  term  "task"  also  has  various  connotations.  We  are  concerned 
here  only  with  the  pilot's  task,  which  includes  controlling  the  aircraft  as 
well  as  associated  functions,  not  directly  related  to  controlling  the  aircraft, 
such  as  navigation  and  communications.  A  task  in  the  sense  that  it  is  used  in 
handling  qualities  evaluations  is  defined  as  "the  actual  work  assigned  a  pilot 
to  be  perfoimed  irt  completion  of.  Or  as  representative -of,  a  designated^ flight 
segment."  In  being  representative  of  a  flight  phase,  for  exajiq)le,  the  impor¬ 
tant  pi  lot- vehicle  functions  required  to  fulfill  the  operational  requirements 
for  that  flight  phase  would  be  represented  in  the  task.  Use  of  "task"  and 
"mission"  differs  then,  in  that  a  task  represents  what  the  pilot  is  actually 
asked  to  do  (as  in  a  simulation  task)  while  a  mission  refers  to  all  opera¬ 
tional  requirements  the  pilot-vehicle  combination  must  be  able  to  accomplish 
if  the  "intended  use"  of  the  aircraft  is  to  be  fulfilled. 

It  is  convenient  to  consider  the  complete  task  to  be  composed  of  (1)  the 
control  task,  and  (2)  auxiliary  tasks.  The  control  task  requires  actuation 
of  the  principal  controls  and  the  selectors  as  required.  The  auxiliary  tasks 
involve  the  pilot  in  actions  other  than  direct  control  of  the  aircraft. 

Flight  phase.-  The  terms  "flight,"  "flight  phase,"  and  "flight  subphase" 
denote  the  flight  profile  of  an  aircraft  and  its  subdivision  into  convenient 
segments.  The  delineation  of  aircraft  role  and  the  division  of  a  complete 
flight  into  discrete  segments  for  more  definitive  examination  is  illustrated 
in  figure  2.  Representative  examples  of  what  is  meant  by  aircraft  role,  com¬ 
plete  flight,  flight  phase,  and  subphase  are  given  in  this  figure  as  well  as 
in  appendix  A.  A  subphase  is  defined  as  "that  part  of  a  flight  phase  having 
a  single  objective,  and  a  single  configuration  or  change  in  configuration." 

A  subphase  evaluation  then  would  provide  a  direct  correlation  between  a 
specific  set  of  stability  and  control  parameters  and  pilot  rating. 

In  summary  then,  the  role  of  an  aircraft  defines  its  intended  use  only 
in  a  general  sense.  The  mission  delineates  this  use  in  terms  of  specific 
objectives,  that  is,  the  required  operations  of  the  pilot-vehicle  combination. 
The  task  delineates  those  aspects  of  the  mission  that  are  work  assigned  to  the 
pilot. 


Methods  of  Determining  Aircraft  Handling  Qualities 

The  relationship  between  stability  and  control  parameters  and  the  degree 
of  suitability  of  the  airplane  for  the  mission  may  be  examined  by: 
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Figure  2.-  The  relationship  between  role,  flight  segments,  and  task. 

Theoretical  analysis 

2.  Experimental  performance  measurement 
-  a.  Pilot  input 

b.  Pi lot- vehicle  output 

3.  Pilot  evaluation 

Each  approach  has  an  important  part  in  the  coiiplete  evaluation.  One 
might  ask,  however,  ’’Why  is  the  pilot  assessment  necessary?"  The  answer  must 
consider  the  two  alternatives,  theoretical  analysis  and  performance  measure¬ 
ment.  At  present,  the  applicability  of  the  mathematical  analysis  including 
representation  of  the  human  operator  is  restricted  to  the  analysis  of  speci¬ 
fic  simple  tasks.  Since  the  intended  use  (mission)  is  made  up  of  several 
tasks  and  several  modes  of  pilot-vehicle  behavior,  it  is  difficult  first  to 
describe  accurately  all  modes  analytically,  and,  second  to  integrate  the  qual¬ 
ity  in  the  separate  tasks^into  a  measure  of  overall  quality  for  the  intended 
use.  Theoretical  analysis  is  fundamental  to  the  analytical  prediction  of 
handling  qualities,  but  cannot  adequately  treat  the  complex  interactions  that 
are  now  investigated  by  means  of  e:q)erimental  pilot  evaluation. 

The  attainment  of  satisfactory  performance  in  fulfilling  a  designated 
mission  is,  of  course,  a  fundamental  reason  for  our  concern  with  handling  qual¬ 
ities.  Why,  then,  cannot  the  e:q)erimental  measurement  of  performance  replace 
pilot  evaluation?  Why  not  measure  pilot -vehicle  output  performance  in  the 
intended  use?  Isn't  good  performance  consonant  with  good  quality?  Unfortu¬ 
nately,  the  answer  to  the  latter  question  is  "not  always." 
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A  significant  difficulty  arises  here  in  that,  first,  the  tasks  selected 
for  measuring  performance  may  not  demand  of  the  pilot  all  that  the  real  mis¬ 
sion  demands,  especially  in  terms  of  distractions,  auxiliary  tasks,  ^d 
stress.  In  the  second  place,  pilot  performance  must  also  be  measured  and 
interpreted  so  that  the  pilot-vehicle  performance  can  be  evaluated  correctly. 
The  pilot  is  an  adaptive  controller  whose  goal  (when  he  is  so  instructed;  is 
to  achieve  good  performance.  In  a  specific  task,  he  is  capable  of  attain¬ 
ing  essentially  the  same  performance  for  a  wide  raige  of  vehicle  characteris¬ 
tics,  at  the  expense  of  significant  reductions  in  his  capacity  to  assume  other 
duties  and  to  plan  subsequent  operations.  Significant  differences  in  his  task 
performance  may  not  be  measured  when  very  real  differences  in  mission  suit¬ 
ability  do  exist.  The  pilot 's- performance  must  be  measured  and  analyzed 

properly  to  show  these  differences,  and  one  must  have  a  clear  appreciation  of 
what  constitutes  objectionable  inputs. 

In  the  third  place,  it  is  difficult,  if  not  impossible,  at  the  present 
time  to  measure  all  important  aspects  of  pilot  performance.  Encouraging 
results  have  been  obtained  in  specific  instances  (refs.  5-7)  wherein  good  cor¬ 
relation  has  been  obtained  between  measurements  of  the  physiaat  effort 
exerted  by  the  pilot  (i.e.,  integral  of  pilot  control  displacement,  force, 
etc.)  and  pilot  rating.  In  such  cases,  it  must  be  assumed  that  differences  in 
mental  effort  and  attention  were  not  significant.  Rather  than  attempt  to  pre¬ 
judge  the  influence  of  mental  effort  and  attention,  the  term  workload  is 
defined  to  include  both  mental  and  physical  effort.  The  use  of  modifying 
terms  will  be  necessary  then  when  a  distinction  is  necessary  between  mental 
and  physical  workload. 

The  questions  that  arise  in  using  performance  measurements  may  be 
summarized  as  follows:  (1)  For  what  maneuvers  and  tasks  should  measurements 
be  made  to  insure  fulfillment  of  the  mission  objectives?  (2)  How  do  we  inte¬ 
grate  and  weigh  the  performance  in  several  tasks  to  get  an  overall  measure  of 
quality  if  measurable  differences  do  exist?  (3)  Is  it  necessary  to  measure  or 
evaluate  pilot  workload  for  the  performance  to  be  meaningful?  If  so,  how  are 
these  factors  weighed  with  those  in  (2)?  (4)  What  disturbances  and  distrac¬ 

tions  are  necessary  to  provide  a  realistic  workload  for  the  pilot  while  his 
performance  in  a  specified  task  is  being  measured?  The  d^ficulties  encoun¬ 
tered  in  answering  each  of  these  questions  provide  some  of  the  reasons  why  con 
tinued  reliance  upon' pilot  evaluation  is  necessary  and  why  much  of  the 
detailed  discussion  is  included  in  this  report.  Although  the  i^e  of  measured 
task  performance  is  not  discussed  further  in  this  report,  con'tinued  efforts 
to  measure  and  interpret  both  the  pilot  performance  ^d  the  pilot -vehicle 
performance  should  be  encouraged.  Such  information  is  important  to  the  under¬ 
standing  of  pilot  adaptation  or  "learning  curves,"  and  to  the  interpretation 
of  pilot  evaluation  data. 

Pilot  evaluation  still  remains  the  only  method  of  assessing  the 
interactions  between  pi lot- vehicle  performance  and  total  workload  in  determin¬ 
ing  suitability  of  an  airplane  for  the  mission.  It  provides  a  basic  measure 
of  quality  and  serves  as  a  standard  with  which  pi  lot- airplane  system  theory 
may  be  developed,  performance  measurements  may  be  correlated,  and  significant 
airplane  design  parameters  and  characteristics  may  be  determined  and 
correlated. 
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Pilot  evaluation  data  generally  consists  of:  (1)  the  pilot  rating,  or 
shorthand  representation  of  the  flying  characteristics  as  they  relate  to  mis¬ 
sion  accomplishment,  and  (2)  the  pilot  comments  that  identify  those 
characteristics  that  interfere  with  the  intended  use. 

A  pilot  rating  is  a  portion  of  the  technical  report  of  the  evaluator,  and 
is  the  overall  summation  of  the  suitability  of  the  vehicle  for  the  specified 
use.  The  pilot  rating  scale  is  then  a  systematic  means  of  denoting  the  qual¬ 
ity  of  the  pilot-vehicle  combination  in  the  accomplishment  of  its  intended 
purpose. 


EARLY  RATING  SCALES 


In  early  handling  qualities  research,  each  investigator  tended  to  develop 
a  rating  scale  peculiar  to  the  needs  of  his  specific  program,  or  to  modify  an 
existing  one.  With  experience,  certain  pilot  rating  scales  proved  successful 
and  stimulated  further  interest  in  them.  Two  such  early  sculss  are  discussed 
here. 


In  reference  8  th€#original  Cooper  Scale  (fig.  3)  proposed  the  basic 
framework  of  boundaries  that  is  still  the  foundation  of  most  pilot  rating 
scales,  including  the  presently  proposed  one.  This  framework  involves  several 
grades  of  quality  pertaining  to  the  intended  use  of  the  vehicle. 

Aa:eptable  and  Satisfactory  -  sufficiently  good 

Acceptable  but  Unsatisfactory  -  not  sufficiently  good,  but  still  usable 

Unacceptable  -  not  usable  for  mission 

Uncontrollable 
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Figure  3.-  Original  Cooper  Rating  Scale. 
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Important  contributions  of  the  early  scales  were  to  emphasize  pilot  acceptance 
of  handling  qualities  with  respect  to  the  intended  use  by: 

a.  Encouraging  investigators  to  define  adequately  the  program  objectives 
and  the  intended  use  of  the  aircraft. 

b.  FirmTy  establishing  within  the  scale  the  acceptable-unacceptable,  and 
satisfactory -unsatisfactory  boundaries . 

c.  Creating  a  logical  basis  that  would  enable  the  pilot  to  express  his 
assessment  accurately  and  consistently. 

The  original  Cooper  Scale  included  too  many  different  concepts.  The 
introduction  of  stability  augmentation  and  undefined  failure  modes,  of  normal 
and  of  emergency  operation,  and  the  separation  of  the  landing  task  from  the 
primary  mission  led  to  ambiguous  interpretations. 

The  CAL  scale  of  figure  4  was 
developed  primarily  because  the 
Cooper  Scale  was  confusing  to  some  in 
that  it  could  be  interpreted  as  intro¬ 
ducing  an  alternate  mission  concept. 
Separate  boundaries  were  shown  in  the 
Cooper  Scale  for  normal  operation  and 
for  an  undefined  emergency  condition. 
By  removing  this  doubt  of  mission 
completion  in  the  adjective  descrip¬ 
tions  in  the  acceptable  range,  as 
well  as  removing  all  consideration  of 
an  alternate  mission  from  the  scale 
itself,  the  CAL  scale  clarified  this 
situation.  However,  the  very  simple 
descriptions  of  the  CAL  scale  are  not  considered  particularly  helpful  by  many 
pilots  and  the  dual  use  of  "fair"  and  "bad"  was  confusing  to  some. 

Having  now  identified  certain  deficiencies  and  objections  to  two  rating 
scales  and  clarified  some  of  the  terminology,  we  are  in  a  position  to 
construct  a  revised  scale. 


r 
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Adjective  description 

Numerical 

-  Category 

within  category 

rating 

Acceptable 

Exceilent 

1 

ond 

Good 

2 

satisfactory 

Fair 

3 

Acceptable 

Fair 

4 

but 

Poor 

5 

unsatisfactory 

•  Bod 

6 

Bod* 

7 

Unocceptable 

Very  bad** 

8 

Dongerous* 

9 

^  Unflyable 

Unflyoble 

10  J 

^Requires  major  portion  of  pilot's  ottention 
**Con  troll  able  only  with  a  minimum  of  cockpit  duties 
^Aircraft  just  controllable  with  complete  attention 


Figure  4.-  CAL  Rating  Scale. 


REVISED  PILOT  RATING  SCALE 


Major  Categories 

Category  selection.-  The  Cooper  and  CAL  Rating  Scales  and  the  revised 
scale  proposed  in  references  land  2  have  the  same  basic  structure.  The  major 
categories  are  identified  as  "satisfactory"  and  "unsatisfactory,"  or  "accept¬ 
able,"  "unacceptable,"  and  "uncontrollable."  Such  terms  relate  only  to  the 
individual  'pitot's  assessment  of  quality  relative  to  the  intended  use  of  the 
aircraft.  The  intention  has  been  to  encourage  the  pilot  to  make  the  iii5)ortant 
decisions  identified  with  these  terms.  These  categories  are  systematically 
arranged  so  that  the  pilot  can  choose  between  two  alternatives  that  lead  to 
the  proper  category. 
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This  structure  is  illustrated  by 
the  flow  chart  in  figure  5,  which 
enables  one  to  trace  the  series  of 
dichotomous  decisions  the  pilot  makes 
in  arriving  at  the  final  rating.  As 
a  rule,  the  first  decision  is  fairly 
obvious .  Are  the  handling  qualities 
controllable  or  uncontrollable?  To 
determine  whether  this  decision 
applies  throughout  the  task  or  flight 
phase  and  in  context  of  the  defined 
role  may  not  be  so  obvious. 


Figure  S.-  Sequential  pilot-rating  decisions.  If  the  vehicle  is  uncontrollable, 

it  is  rated  10.  If  it  is  control¬ 
lable,  the  second  decision  is  whether  it  is  acceptable  or  unacceptable.  If 
unacceptable,  the  rating  7,  8,  or  9  will  be  selected  (rating  10  has  been 
excluded  by  the  "controllable"  answer  to  the  first  decision).  If  it  is  accept¬ 
able,  the  third  decision  is  whether  it  is  satisfactory  or  unsatisfactory.  If 
unsatisfactory,  the  rating  4,  5,  or  6  will  be  selected;  if  satisfactory,  the 
rating  1,  2,  or  3  will  be  selected. 


Category  definitions.-  The  dichotomous  decisions  outlined  in  the  previous 
paragraph  logically  lead  to  four  categories  of  quality,  some  of  which  have 
been  difficult  to  describe  accurately  by  a  single  adjective.  There  are  objec¬ 
tions  to  the  designation  "unsatisfactory  but  acceptable"  and  "unacceptable  but 
controFlable,"  but  no  simple  adjective  descriptions  for  these  categories  have 
been  found  that  will  satisfy  everyone.  In  spite  of  strong  emphasis  as  to  the 
need  to  establish  clearly  that  the  terms  are  related  to  the  pilot  's  aum 
assessment  of  acceptability  and  not  to  a,ny  existing  standards,  specifications, 
or  other  Acceptance  Criteria,  objection  to  "acceptable"  and  "unacceptable"  in 
the  scale  remains. 


Regardless  of  the  terms  used  in  a  revised  rating  scale,  it  is  necessary 
to  clarify  the  intended  meaning  of  satisfactory,  unsatisfactory,  acceptable, 
tolerable,  unacceptable,  controllable,  and  uncontrollable.  Let  us  examine, 
first,  what  is  meant  by  "controllable,"  even  though  it  has  caused  less  diffi¬ 
culty  than  the  other  terms.  To  control  is  to  exercise  direction  of,  to  com¬ 
mand,  or  to  regulate.  The  determination  as  to  whether  the  airplane  is 
controllable  must  be  made  within  the  framework  of  the  defined  mission  or 
intended  use.  An  example  of  the  considerations  of  this  decision  would  be  the 
evaluation  of  fighter  handling  qualities  during  which  the  evaluation  pilot 
encounters  a  situation  in  which  he  can  maintain  control  only  with  his  complete 
and  undivided  attention.  The  vehicle  is  "controllable"  in  this  situation  in 
the  sense  that  the  pilot  can  maintain  control  only  by  restricting  the  tasks 
and  maneuvers  he  is  called  upon  to  perform  and  by  giving  the  configuration  his 
undivided  attention.  However,  for  him  to  answer,  "Yes,  it  is  controllable  in 
the  flight  phase  (or  task),"  he  must  be  able  to  retain  control  in  all  mission 
oriented  tasks  and  other  required  operations  without  sacrificing  effort  and 
attention  to  his  overall  duties. 
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Consider  now  the  meaning  of  acceptable,  the  usage  of  which  has  been  the 
subject  of  some  controversy,  and  the  alternate  suggested  term  "tolerable." 

The  dictionary  shows  "acceptable"  to  mean  that  a  thing  offered  is  received 
with  a  consenting  mind;  "unacceptable"  to  mean  that  it  is  refused  or  rejected. 
Acceptable  means  that  the  flight  .phase  (or  task)  can  be  acconqjlished;  it 
means  that  the  evaluation  pilot  would  agree  to  use  it  for  the  designated  role; 
that  such  deficiencies  as  may  exist  can  be  endured  or  tolerated.  Use  of  the 
term  "acceptable"  does  not  say  how  good  it  is,  but  it  does  say  the  pilot  con¬ 
siders  it  good  enough  for  the  intended  use.  With  these  characteristics,  the 
flight  phase  (or  task)  can  be  accomplished  with  adequate  precision.  The  task, 
for  example,  may.  be  accomplished  with  considerable  effort  and  concentration 
on  the  part  of  the  pilot,  but  the  level  of  workload  required  to  achieve  this 
performance  is  tolerable  and  not  unreasonable  in  context  with  the  intended 
use.  By  the  same  token,  "unacceptable"  does  not  necessarily  mean  that  the 
designated  flight  phase  (or  task)  cannot  be  accomplished;  it  does  mean  that 
the  necessary  performance  cannot  be  achieved  or  that  the  effort,  concentration, 
and  workload  required  are  of  such  magnitude  that  the  evaluation  pilot  rejects 
the  aircraft  for  this  phase  of  its  intended  use. 

Consider  now  a  definition  of  "satisfactory."  The  dictionary  defines  this 
as  adequate  for  the  purpose,  of  a  kind  to  meet  all  requirements  or  expecta¬ 
tions.  A  pilot's  definition  of  satisfactory  might  be  that  it  isn't  neces¬ 
sarily  perfect,  or  even  good,  but  it  is  good  enough  that  he  wouldn't  ask  that 
it  be  changed.  It  meets  a  standard;  it  has  sufficient  goodness;  it's  of  a 
kind  to  meet  all  pilot  demands  for  the  intended  use. 

Unsatisfactory  inplies  that  there  is  insufficient  goodness  to  meet  all 
pilot  demands;  that  it  has  deficiencies  and  objectionable  characteristics 
which  he  feels  should  be  corrected.  Unsatisfactory  includes  all  that  is  not 
satisfactory;  just  as  acceptable  includes  all  that  is  not  specifically  refused 
as  unacceptable.  A  specific  category  is  then  that  which  is  "unsatisfactory 
but  acceptable."  This  category  has  previously  been  referred  to  by  the  shorter 
term  "acceptable"  but  could  also  be  referred  to  as  "tolerable"  in  the  sense, 
"capable  of  being  borne,  supportable,  bearable."  As  a  result  of  the  possibil¬ 
ity  of  the  misinterpretation  in  the  use  of  "acceptable"  it  has  been  found 
preferable  to  use  "tolerable"  rather  than  "acceptable"  in  the  rating  scale. 

Thus,  the  quality  is  either: 

a.  Satisfactory  -  good  enough  without  improvement,  and,  therefore,  of 
the  best  category,  or 

b.  Unsatisfactory  but  tolerable  -  just  good  enough,  adequate  for  the 
purpose  but  improvement  desirable,  and,  therefore,  of  the  next  best  category, 
or 

c.  Unacceptable  to  the  pilot  -  not  suitable  for  the  purpose  but  still 
controllable,  and  in  the  third  category,  or 

d.  Uncontrollable  -  imacceptable  for  the  purpose  and  of  the  poorest 
quality  and  in  the  fourth  category. 
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Inasmuch  as  only  four  categories  of  quality  are  needed  to  identify  and 
describe  handling  qualities,  it  is  possible  to  simplify  the  dichotomous 
decision  process  illustrated  in  figure  5  and  eliminate  ''acceptability"  terms . 

By  considering  the  following  three  decisions  the  pilot  will  arrive  at  one 
of  the  four  categories  previously  discussed: 


1.  Is  the  vehicle  controllable? 

2.  Is  adequate  performance  attainable  with  a  tolerable  workload? 

3.  Is  the  vehicle  satisfactory  without  improvement? 


ADEQUACY  FOR  SELECTED  TASK  OR  REQUIRED  OPERATION 


•  Desired  or  cleoriy  adequate 
performance  is  attoinable. 

•  No  significant  deficiencies. 

•  Good  enough  without  improvement. 


•  Adequate  performance  is  attainable 
but  requires  pitot  compensation. 

•  Has  deficiencies  which  warrant 
improvement. 


•  Performonce  is  inadequate  even  with 
maximum  tolerable  pilot  compensation. 

•  Has  major  deficiencies  which  require 
improvement. 


•  Control  will  be  lost  during  some 
portion  of  required  ope^ion. 

•  Improvement  is  mondatory. 


Figure  6.-  Major  category  selection  and  definition. 


Being  able  to  designate  a 
category  by  a  single  adjective  was 
considered  less  important  than  that 
the  definition  of  each  category  be 
precise.  The  rating  scale  now  pro¬ 
posed  is  therefore  based  on  the  cate¬ 
gory  definitions  provided  in  figure  6. 
These  are  reasonably  concise  and 
emphasize  the  pilot's  decisions 
involved  in  category  selection. 

Fimdamental  to  this  selection  is 
the  consideration  of  whether  adequate, 
performance  is  attainable  in  the 
selected  task  or  required  operation 
(mission)  and  whether  or  not  defici¬ 
encies  are  present  which  require 
pilot  condensation. 

Assessment  of  task  performance 
and  pilot  workload  then  enables  the 
pilot  to  decide  whether  the  handling 
qualities  are  good  enough  without 
indrovement ,  have  deficiencies  for 
which  indrovement  is  desired,  or  have 
deficiencies  for  which  improvement  is 
essential . 


Individual  Ratings 

The  coBdlete  revised  rating  scale^  in  figure  7  includes  further 
subdivisions  of  quality  within  three  of  the  four  major  categories  with  appro¬ 
priate  descriptions  for  each  numerical  rating  to  define  quality  differences. 
These  allow  the  pilot  a  sufficient  range  of  handling  quality  descriptions  for 
most  situations.  Repetition  of  descriptive  terms  from  the  category  defini¬ 
tions  has  all  but  been  eliminated  in  the  individual  rating  descriptors. 

^Small  and  large  copies  of  figure  7  are  available.  Send  requests  to 
Technical  Information  Division,  Ames  Research  Center,  Moffett  Field,  Calif., 
94035 . 
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Figure  7.-  Handling  Qualities  Rating  Scale. 


Considered  desirable  in  simplifying  the  scale  for  the  experienced  user,  this 
arrangement,  however,  makes  it  mandatory  that  any  user  understand  and 
the  category  definitions  and  make  the  decisions  outlined  on  the  left.  TOe 
important  "boundary"  decisions  between  pilot  ratings  of  3  and  4,  6  and  7,  or 
9  and  10  cannot  be  made  by  reference  to  the  individual  rating  descriptors 
alone.  It  is  emphasized  that  these  descriptions  supplement  the  sequential 
decisions  that  lead  the  evaluation  pilot  to  the  particular  category  within 
whidi  the  descriptions  of  the  individual  ratings  are  given.  It  was  considered 
fundamental  to  a  good,  easily  applied  scale  that  the  descriptions  be  both 
brief  and  general.  Key  words  and  phrases  were  sought  that  would  easily  be 
understood  and  yet  sufficiently  definitive  so  that  each  rating  would  be 
clearly  separated  from  every  other  rating.  The  following  paragraphs  discxiss 
and  explain  some  of  the  factors  related  to  the  rating  scale. 

Performance  and  workload.-  Consideration  was  given  to  describing  quality 
in  terms  of  both  performance (precision  of  aircraft  control)^  and  workload 
(effort  and  attention)^  for  eacJi  numerical  rating.  Upon  closer  examination, 

2when  applied  to  handling  qualities,  the  term  "performance"  alone  is 
intended  to  mean  the  precision  of  aircraft  control  attained  by  the  pilot,  that 
is,  the  pilot- vehicle  performance.  The  term  "workload"  is  intended  to  convey 
the  amount  of  effort  and  attention,  both  physical  and  mental,  that  the  pilot 
must  provide  to  attain  a  given  level  of  performance.  "Pilot  performance"  may 
be  used  to  describe  the  measure  of  physi-oaZ  workload  (effort  and  attention) 
used  in  performing  a  task. 
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however,  it  was  concluded  that  these  factors  were  so  interdependent  and 
subject  to  tradeoff  by  a  pilot  that  it  would  be  impossible  to  define  non¬ 
conflicting  individual  ratings .  Precision  of  control  could  not  be  defined 
independently  of  the  amount  of  effort  and  attention  required  of  the  pilot  for 
any  task. 

The  best  category  is  defined  as  desired  or  otearly  adequate  performance 
in  association  with  a  satisfactory  level  of  pilot  workload.  The  second  best 
category  is  defined  by  achievement  of  adequate  performance  although  the  pilot 
is  required  to  conqjensate  for  deficiencies  by  increasing  his  workload.  It  is 
tolerable  to  him  but  he  desires  improvement.  The  third  category  involves 
major  deficiencies  because  the- pilot~-fi-nds  it  impossible  .to  achieve  adequate 
performance  even  though  his  attempt  to  coii5)ensate  increases  the  total  workload 
to  the  maximum  tolerable  in  context  of  the  task  or  mission.  Unoontrolldble 
obviates  further  concern  with  performance,  but  as  in  all  previous  categories, 
must  also  be  considered  in  the  context  of  the  selected  task  or  required 
operation  (mission) . 

Compensation.-  "Pilot  compensation"  as  used  in  the  scale  is  intended  to 
indicate  that  the  pilot  must  increase  his  workload  to  improve  aircraft  perfor¬ 
mance.  It  relates  the  pilot's  difficulty  in  completing  a  task  with  the 
precision  required  for  that  task.  Stated  another  way,  it  is  the  measure  of 
additional  pilot  effort  and  attention  required  to  maintain  a  given  level  of 
performance  in  the  face  of  less  favorable  or  deficient  characteristics.  The 
total  workload  is  then  comprised  of  the  workload  due  to  compensation  for  air¬ 
craft  deficiencies  plus  the  workload  due  to  the  task. 

Referring  to  figure  6,  we  see  that  it  is  really  only  necessary  to  define 
that  which  constitutes  adequate  performance  when  answering  the  second  question 
The  precision  of  control  required  for  any  task  is  most  easily  defined  in  terms 
of  the  end. result  obtained.  For  example,  the  approadi  performance  can  be  spec 
ified  in  t.erms  of  the  threshold  "gate,"  and  landing  performance  is  measured  in 
terms  of  vertical  velocity  at  touchdown  and  dispersion  about  the  intended 
touchdown  point.  The  necessity  for  holding  airspeed,  altitude,  flight  path, 
or  other  parameters  within  specified  limits  throughout  each  flight  phase  or 
subphase  may  also  be  specified,  but  the  cause  and  effect  of  occasionally 
exceeding  the  limits  may  require  pilot  interpretation.  The  pilot  must  balance 
both  aspects  of  performance  -  maintaining  precision  of  control  and  the  end 
result  -  against  his  own  effort  and  attention  in  arriving  at  a  rating. 

The  category  definitions  of  figure  6  recognize  the  interaction  between 
performance  achieved  and  the  pilot  workload  required  to  compensate  for  defi¬ 
cient  characteristics.  In  the  best  category,  no  significant  compensation^  in 
terms  of  added  workload,  is  required  by  the  pilot  to  achieve  adequate  perfor¬ 
mance.  In  fact,  adequate  performance  is  clearly  achievable  and  it  may  just 
as  easily  be  possible  to  attain  some  higher  level  of  performance,  such  as 
might  be  designated  broadly  as  desired  performance.  From  the  definition,  it 
may  be  deduced  that  desired,  or  at  least  clearly  adeqviate,  precision  of  con¬ 
trol  can  be  obtained  with  relative  ease  or  a  low  level  of  effort  and  atten¬ 
tion.  With  deficiencies  characterized  by  the  second  category,  considerably 
more  pilot  effort  is  required  and  desired  performance  is  not  necessarily 
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obtained.  Now,  however,  a  lower  level  of  handling  perform^ce,  but  one 
adequate  for  the  purpose,  can  be  achieved  as  long  as  the  pilot  will  incre^e 
his  workload,  even  to  the  maximum  tolerable  for  the  selected  task.  If  major 
deficiencies  exist,  adequate  precision  of  control  would  require  an  excessive 
level  of  pilot  workload  and  even  the  maximum  tolerable  pilot  workload  will  no 
enable  achievement  of  the  level  of  performance  that  is  considered  adequate 
for  the  selected  task.  In  this  category,  the  additional  deficiencies  that 
increase  demands  on  the  pilot  can  only  be  expected  to  result  in -vruideqiu^e 
performance,  and  the  primary  question  becomes  one  of  controllability, 
manner  in  which  precision  of  control  may  be  traded  off  against  pilot  effor 
and  attention  is  complex  and  defies  explicit  definition  for  all  flight  phases 

or  specified  tasks'.  However;  such  compensation  is  available  to  the  pilot  and 

is  probably  involved  in  the  evaluation  of  handling  qualities. 


Failures  or  emergency  operation.-  The  revised  scale  contains  no  reference 
to  failure  considerations  or  emergency  operation.  In  effect,  this  means  that 
the  pilot  need  not  always  concern  himself  during  an  evaluation  with  the  prob¬ 
ability  of  a  failure,  nor  with  the  length  of  time  he  might  be  faced  with 
deficient  handling  qualities .  The  time  (duration.)  during  which  particular 
characteristics  must  be  coped. with  is  inherently  defined  by  the  role  and  the 
flight  phase  or  task  being  evaluated.  For  operation  based  on  a  normal  air¬ 
craft  state,  the  duration  can  be  based  explicitly  on  the  aircraft  role.  If  a 
failure  occurs,  however,  it  may  be  necessary  to  consider  alternatives  like 
aborting  the  flight  or  changing  the  flight  plan  to  permit  retreat  to  a  more 
favorable  flight  condition;  in  either  case,  the  length  of  time  that  the  criti¬ 
cal  flight  condition  must  be  tolerated  might  be  shortened.  Unless  such 
altemSives  are  spelled  out  in  the  task  definition,  however,  the  pilot  must 
always  treat  a  failure  state  as  having  to  be  coped  with  for  the  duration  of 
the  task  or  flight  phase,  depending  on  which  is  being  rated.  Normal  and 
emergency  states  then  will  require  separate  evaluations. 


There  are  environmental  conditions  as  well  as  minor  failures  which  can 
limit  aircraft  operation  without  being  identified  as  an  emergency.  Excessive 
turbulence  or  crosswind  could  preclude  attempting  a  landing  and  thus  require 
that  the  flight  plan  be  altered  accordingly.  Whenever  limitations  to  the 
operation  are  indicated  or  accepted  in  lieu  of  specifying  a  critical  task  this 
fact  should  be  clearly  noted. 


Pilot  skill.-  The  pilot  effort  and  attention  required  for  a  given  task  or 
flight  phase  will,  of  course,  depend  somewhat  upon  individual  pilot  skill  and 
state  of  training.  In  the  early  versions  of  the  revised  scale,  the  term 
"skill"  was  used  in  the  8  and  9  rating  descriptions  in  consideration  of  the 
role  of  pilot  skill  in  the  degree  of  controllability  which  could  be  achieved. 
Further  consideration,  however,  has  led  in  the  8  and  9  rating  descriptions 
to  the  substitution  of  phrases  reflecting  pilot  workload  required  to  even  con¬ 
tinue  the  task  and  the  relative  difficulty  in  maintaining  control,  in  order  to 
retain  a  consistency  and  compatibility  with  the  rest  of  the  scale.  In  this 
manner,  pilot  skill  becomes  a  consideration  of  the  total  program  for  which  the 
evaluation  is  being  made. 
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It  is  the  opinion  of  the  authors  that  as  a  general  rule  each  pilot  should 
judge  the  suitability  of  any  set  of  airplane  characteristics  in  terms  of  his 
own  skill  and  training,  and  in  terms  of  the  required  operations  and  circum¬ 
stances  as  defined  in  the  experiment.  The  effects  of  differing  skills  should 
be  determined  from  the  results  obtained  from  evaluation  pilots  of  different, 
but  representative,  levels  of  experience  and  training.  Exceptions  to  this 
general  rule  have  occurred,  however,  when  the  research  or  development  test 
pilot  is  asked  to  evaluate  handling  qualities  with  respect  to  his  understand¬ 
ing  of  the  lowest  degree  of  skill  and  training  existent  in  a  group  of 
operational  pilots. 

Operating  margins  and  safety.-  The  question  of  safety  cannot  be  separated 
from  handling  qualities  because  the  precision  with  which  certain  parameters 
are  controllable  determines  whether  available  safety  margins  are  apt  to  be 
exceeded.  The  margins  provided,  as  well  as  the  consequence  of  exceeding  them, 
will  certainly  influence  the  effort  and  attention  which  a  pilot  applies.  Oper¬ 
ating  margins  with  respect  to  such  parameters  as  airspeed,  angle  of  attack, 

Mach  number,  and  altitude  then  represent  the  constraints  that  define  required 
precision  of  control,  and  thereby  influence  the  rating  selection.  Operating 
margins  and  safety  should  be  recognized  as  constraints  upon  handling  qualities 
that  are  inextricably  related  to  performance.,  and  workload. 


Considerations  Associated  With  the  Structure  of  the  Revised  Scale 

.  Why  a  10-point  scale?-  In  discussing  the  revised  scale,  one  question 
that  might  be  anticipated  concerns  the  number  of  individual  ratings  the  scale 
defines^  Most  simply,  the  number  is  related  to  the  four  categories  already 
selected.  Separating 'each  of  fhe  upper  three  categories  into  three  subratings 
appears  to  provide  an  adequate  spread  for  pilot  use.  Additional  ratings  in 
the  fourth  category  (uncontrollable)  would  not  appear  to  be  of  general  value. 

A  change  in  the  nuidser  of  individual  ratings  for  each  category  was  not  deemed 
necessary  or  desirable  when  considered  in  light  of  the  large  amount  of  e^er- 
ience  with  the  previous  scales. 

Identifying  the  revised  rating.-  The  oft-proclaimed  criticism  that  the 
scale  should  start  with  10  and  progress  to  1  instead  of  from  1  to  10  may  be 
valid,  but  there  are  also  examples  that  support  the  1  to  10  logic.  We  are 
reluctant  at  this  point  to  suggest  a  change  siii5)ly  because  of  the  widespread 
use  of  3-1/2  and  6-1/2  boundaries.  To  now  reverse  these  would  likely 
introduce  considerable  confusion  and  would  not  necessarily  have  long-range 
benefits. 

In  proposing  a  revised  scale,  it  was  recognized  that  some  confusion  might 
result  from  continued  use  of  the  same  numerical  scale  that  has  been  identified 
with  both  the  Cooper  and  CAL  Scales.  In  references  1  and  2,  the  authors  pro¬ 
posed  a  modified  identification  system  using  letters  A  and  U  in  conjunction 
with  the  numbers  1-9.  Subsequent  experience  and  comment  indicated  this  to 
be  unnecessary,  as  long  as  the  rating  numerals  are  associated  explicitly  with 
the  particular  scale  used. 
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Linearity.-  This  is  a  desirable  diaract eristic  of  any  scale.  A 
temperature  scale  is  linear  with  heat  added  for  a  material  with  constant  spe¬ 
cific  heat,  in  that  the  temperature  rise  per  vmit  quantity  of  heat  added  is 
the  same  throughout  the  scale.  Temperature  is  a  normal  and  useful  scale  asso¬ 
ciated  with  comfort.  Even  though  temperature  may  be  quite  linear  with  heat 
added,  comfort  is  not  linear  with  ten5)erature .  With  what  should  the  pilot  rat¬ 
ing  scale  be  linear?  Since  it  is  purported  to  measure  quality,  it  should  then 
be  linear  with  the  added  quality  of  the  pilot-vehicle  combination  in  that  the 
change  in  pilot  rating  per  unit  quality  addition  should  be  the  same  throughout 
the  rating  scale.  The  rating  scale  may  possibly  have  this  characteristic,  but 
to  demonstrate  that  the  scale  is  indeed  linear  would  require  an  independent 
measure  of  qixality  that  does  not  presently  exist.  Since  the  basic  merit  of 
the  scale  is  not  significantly  affected  by  the  lack  of  demonstrated  linearity, 
this  factor  has  not  been  considered  further.  McDonnell  in  reference  4 
describes  a  study  that  establishes  a  correlation  between  the  Cooper  Scale, 
the  revised  scale  of  references  1  and  2,  and  a  linear  scale. 

Ordinal  versus  interval  scale.-  An  interval  scale  is  desirable,  but  the 
proposed  pilot  rating  scale  cannot  be  shown  to  be  an  interval  scale.  The 
authors  have  accepted  it  as  being  ordinal.  It  is,  however,  primarily  an  abso¬ 
lute  scale  rather  than  a  relative  one.  The  pilot  rating  is  given  for  a  con¬ 
figuration  in  the  context  of  its  acceptability  to  the  pilot  for  the  specified 
flight  phase  (or  task)  and  not  in  terms  of  its  goodness  with  respect  to  a  con¬ 
figuration  already  evaluated.  Fortunately ,  the  concentration  and  effort 
required  in  performing  each  evaluation  tends  to  suppress  in  the  pilot’s  mem¬ 
ory  the,  diaracteris tics  of  preceding  configurations,  enabling  him  to  consider 
objectively  each  configuration  on  its  own  merits  for  the  required  operations 
without _contin\ially  making  paired  con^arisons .  Pilots  are  reluctant  to  rate 
something  as  excellent  or  optimum  for  fear  that  a  subsequent  configuration 
will  be  better  than  anything  they  considered  possible. 

Words  versus  numbers.-  The  basic  structure  of  the  rating  scale  is 
completely  dependent  on  words  and  their  explicit  definitions .  The  numeral 
associated  with  the  evaluator’s  final  decision  is  an  expedient,  a  shorthand 
symbol.  One  risk  associated  with  a  numerical  scale  is  that  engineers  will 
attempt  to  treat  the  pilot  rating  data  with  mathematical  operations  that  are 
rigorously  applicable  only  to  a  linear  interval  scale.  Although  some  insight 
is  sometimes  gained,  analysis  of  specific  pilot  rating  data  should  not  be 
totally  dependent  on  such  mathematical  operations. 

Differing  standards  of  acceptance  for  the  sjame  mission.-  One  difficulty 
that  has  arisen  in  the  use  of  pilot  rating  data  can  be  illustrated  by  the  fol¬ 
lowing  example.  In  an  evaluation  program  for  the  landing  approach  flight 
phase  of  a  commercial  air  transport  the  role  and  mission  were  carefully 
defined,  the  program  was  run,  and  the  results  were  reported.  One  of  the  eval¬ 
uation  pilots,  an  airline  pilot,  subsequently  remarked  that  his  airline  would 
not  accept  any  airplane  with  worse  than  a  4-1/2  rating  for  the  landing 
approach.  However,  using  the  generally  accepted  interpretation  of  the  rating 
scale,  this  pilot  had  said,  as  an  evaluation  pilot,  that  aircraft  with  worse 
characteristics  whidi  he  rated  at  5  and  6  could  still  be  considered  acceptable 
for  the  intended  use.  In  the  first  case  he  was  reflecting  the  viewpoint  of 
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airline  management  and  not  speaking  from  the  pilot's  standpoint.  It  must  be 
recognized  that  the  final  determination  of  overall  acceptability  of  an  air¬ 
craft  for  a  given  role  may  be  influenced  by  factors  other  than  just  the 
piloting  rating  of  the  handling  qualities. 

In  deciding  what  to  buy,  any  customer  considers  what  he  will  get  in  terms 
of  how  much  it  will  cost.  It  is  easy  to  envision  similar  decisions  being  made 
to  buy  only  that  which  is  above  the  3-1/2  boundary.  And  similarly,  one  can 
envision  a  .reluctant  decision  to  buy  as  low  as  a  6  rating,  or  to  even  accept 
a  7  or  8  rating  (if  the  rating  applied  to  a  flight  condition  or  aircraft  fail¬ 
ure  state  of  low  probability),  but  only  if  all  other  possibilities  for  pur¬ 
chase  of -a  better  aircraft  had  been  excluded-.- -  The  basic,  pilot  rating,  data  on 
which  these  decisions  are  based,  however,  must  be  strictly  mission-oriented  if 
the  subsequent  quality  versus  cost  decisions  are  to  be  meaningful.  That  is  to 
say,  the  role  must  be  ijnderstood  and  each  flight  phase  adequately  defined  by 
the  piloting  task(s)  provided. 


EXPERIMENTAL  USE  OF  PILOT  RATING  OF  HANDLING  QUALITIES 


In  previous  usage  of  pilot  rating  scales,  too  little  attention  has  been 
given  to  defining  just  what  the  pilot  was  rating  and  how  the  data  were  to  be 
used.  In  defining  the  experiment  and  in  reporting  the  results,  it  became 
apparent  that  (1)  the  term  "handling  qualities"  requires  a  tighter,  less  ambig¬ 
uous  definition,  (2)  an  accounting  is  required  for  certain  factors  in  addition 
to  stability  and  control,  (3)  a  clearer  understanding  is  needed  of  the  differ¬ 
ence  between  mission  and  task  definition,  (4)  certain  considerations  relative 
to  the  pilots  assessment  criteria  must  be  understood,  and  (5)  a  simulation 
situation  requires  special  definition  and  consideration  of  the  use  of  the 
results  and  of  the  need  for  pilot  extrapolation. 

In  the  following  paragraphs  attention  is  devoted  to  certain  aspects  of 
the  design  and  execution  of  handling  qualities  experiments  that  are  considered 
particularly  important  to  the  production  of  good  data  and  the  further 
clarification  of  the  aforementioned  problems. 


Program  Development 

Fundamental  to  any  handling  qualities  program  is  a  clear  definition  of 
the  primary  objectives  of  the  program  and  of  the  role  and  mission  of  the  air¬ 
craft.  Next  is  the  designation  of  the  tasks  to  be  used  in  the  course  of  the 
evaluation,  and  what  the  rating  applies  to. 

Program  objective.-  Generally,  it  is  expected  that  the .use  of  the  data 
will  conform  reasonably  well  with  the  objectives  outlined  for  the  program. 

The  primary  objectives  axe  related  to  either  research,  development,  or  accep¬ 
tance  but  there  may  also  be  a  number  of  special  considerations.  (These 
include  whether  the  program  is  exploratory  or  expected  to  be  highly  definitive, 
etc.)  It  is  essential  to  define  whether  stability  and  control  characteristics 
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or  cockpit  interface  elements  are  the  primary  variables  of  investigation  and 
whether  special  conditions  such  as  environmental  disturbances,  dynamic  or 
static  failures,  or  stresses  are  to  be  included  in  the  task  or  considered  in 
the  rating.  Of  particular  importance  will  be  the  selection  of  evaluation 
pilots  and  their  indoctrination  with  respect  to  the  program.  The  particular 
background,  training,  experience,  and  point  of  view  of  the  evaluation  pilots 
selected  for  the  program  may  be  determined  by  the  program  objective  as  well  as 
by  the  intended  use  of  the  resultant  data. 

Mission  description.-  The  explicit  description  of  the  mission  by 
delineation  of  the  "required  operations"  is  probably  the  most  important  con¬ 
tributor'  to  the  objectivity  of  the  pilot  evaluation- data.  The  role  must  be 
carefully  analyzed  and  a  clear  description  and  understanding  reached  between 
the  engineer  and  the  evaluation  pilot  as  to  their  interpretation  of  the 
required  operations.  This  description  must  include: 

a.  What  the  pilot  is  required  to  accoin)lish  with  the  aircraft,  and 

b.  The  conditions  or  circumstances  under  which  the  mission  is  to  be 
conducted. 

Because  of  their  importance  in  specifying  "what  the  pilot  rating  applies 
to"  and  in  providing  definitive  guidance  for  the  pilot  in  his  evaluation,  the 
"required  operations"  may  often  have  to  be  given  in  considerable  detail.  In 
its  siii5)lest  form,  this  consists  of  designating  the  flight  phase  or  subphase(s) 
of  interest,  and  including  such  variations  as  are  considered  critical  and 
representative  of  actual  operations. 

As  an  exaii5)le  of  (b) ,  the  conditions  or  circumstances  might  include 
instrument  or  visual  flight,  type  of  displays  or  controls  in  the  cockpit,  or 
other  input  information  to  assist  the  pilot  in  accomplishing  the  mission,  etc. 
The  environment  in  which  the  mission  is  to  be  accomplished  must  also  be 
defined  and  considered  in  the  evaluation,  and  could  include,  for  example,  the 
presence  or  absence  of  turbulence,  day  versus  night,  the  frequency  with  which 
the  mission  has  to  be  repeated,  the  preparedness  of  the  pilot  for  the  mission, 
and  the  pilot's  level  of  proficiency. 

As  noted  under  the  discussion  of  the  revised  scale,  it  was  recommended 
that  steady-state  failure  considerations,  with  the  attendant  questions  as  to 
probability  of  occurrence,  should  be  removed  from  the  scale.  This  means  that 
separate  ratings  would  be  obtained  for  each  failure  mode.  Steady-state  fail¬ 
ure  modes  would  then  normally  be  considered  as  existing  throughout  the  evalu¬ 
ation  task  or  flight  phase,  unless,  of  course,  a  specific  evaluation  program 
is  designed  to  consider  them  otherwise. 

In  evaluating  the  impact  of  transient  disturbances  caused  by  either 
environmental  conditions  or  system  failures,  the  pilot  is  still  faced  with  the 
basic  decision  of  what  tradeoff  he  will  accept  between  performance  and  work¬ 
load  and  how  long  he  must  cope  with  the  condition. 

High  probability  occurrences,  such  as  low  to  moderate  turbulence,  wind 
'  shear,  and  cross  wind,  can  be  rated  separately,  as  if  occurring  throughout  the 
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task  or  flight  phase,  but  should  be  included  in  a  composite  rating  or  a  flight 
phase  rating  to  convey  the  realistic  situation.  It  is  preferable  to  evaluate 
separately  such  low  probability  occurrences  as  severe  turbulence  so  that  the 
pilot  is  not  asked  to  weigh  probability  of  occurrence  and  generalize  a  compos¬ 
ite  rating  to  the  point  that  it  loses  value.  For  example,  a  pilot  may  prefer 
to  discuss  a  short  duration  occurrence  of  degraded  handling  qualities  in  the 
pilot  comment  data  rather  than  to  make  a  radical  change  in  the  task  rating  or 
flight  phase  rating. 

Such  questions  of  probability  of  occurrence  and  levels  of  disturbances 
must  be  resolved  as  part  of  the  mission  description  in  the  design  of  the  exper¬ 
iment,  with  special  attention  often  being  required  with  respect  to  pilot 
orientation  and  the  reporting  of  results. 


What  the  Rating  Applies  to 

Task  and  flight  phase  terminology.-  For  a  pilot  to  evaluate  the  handling 
qualities  of  an  aircraft,  he  must  maneuver  it  and  otherwise  use  it  for  its 
intended  purpose.  This  purpose  is  given  in  a  general  sense  by  the  role  but 
more  specifically  by  the  designation  of  the  required  operations  or  mission. 
Next,  it  must  be  decided  whether  the  handling  qualities  are  being  evaluated 
for  a  complete  flight,  a  single  flight  phase,  a  subphase,  or  a  specifically 
defined  task.  In  any  event,  a  task  or  series  of  tasks  must  be  specified  that 
will  provide  a  suitable  basis  for  pilot  evaluation.  The  extent  to  which  the 
evaluation  task  selected  represents  all  aspects  of  the  aircraft  mission  will 
depencTupon  several  factors,  the  more  important  of  which  are  program  objective, 
simulation  capability,  and  certain  pilot  considerations.  From  the  definitions 
provided,  it  will  be  possible  to  establish  conventional  procedures  for  refer¬ 
ring  to  task  ratings  in  contrast  to  flight,  flight  phase,  or  subphase  ratings. 

General  considerations.-  To  provide  an  overall  pilot  rating  for  a 
complete  flight  or  aircraft  role  would  likely  involve  so  many  situations  that 
little  of  the  information  would  relate  to  specific  stability  and  control  char¬ 
acteristics.  For  such  a  use,  an  overall  rating  would  be  of  little  value 
unless  reference  were  made  to  the  detailed  comments  provided  by  the  pilot  for 
each  subphase.  A  flight  subphase,  on  the  other  hand,  is  devoted  to  a  single 
objective  and  has  a  single  configuration  or  change  in  configuration.  There¬ 
fore  evaluation  of  a  flight  subphase  would  enable  more  direct  correlation  to 
be  made  between  pilot  rating  and  specific  stability  and  control 
characteristics . 

Should  significant  changes  then  occur  in  either  the  aircraft  state,  the 
cockpit  interface  or  the  aircraft  environment  during  a  given  flight  phase,  it 
might  be  desirable  to  confine  the  pilot  rating  to  a  flight  subphase  defined 
by  a  single  aircraft  state  even  though  the  task  occurs  in  other  subphases  with 
different  aircraft  states. 

If  an  evaluation  is  to  apply  to  a  flight  phase,  the  rating  will  tend  to 
be  weighted  by  the  more  adverse  subphase  characteristics.  On  the  other  hand, 
a  longer  task  representative  of  one  or  more  flight  phases  will  provide  a 
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better  opportunity  for  the  pilot  to  assess  the  workload  involved,  the 
adequacy  of  the  information  displayed,  and  the  effect  of  unanticipated 
environmental  disturbances. 


Short  term  maneuvers,  particularly  if  determined  to  be  the  critical  P^t 
of  the  task,  will  be  most  useful  when  the  effect  of  specific  stability  and 
control  parameters  are  assessed,  but  their  use  to  the  exclusion  of  a  long-term 
mission-oriented  task  can  reduce  both  confidence  in  ^d  the  fidelity  of  the 
handling  qualities  evaluation.  The  inclusion  of  auxiliary  tasks  in  an  eva  na¬ 
tion  task  will  depend  upon  the  program  objective,  but  may  not  be  required  for 
handling  qualities  studies  in  which  the  control  task  is  of  primary  interest. 
Additional  workload  iii5)osed  by  the  auxiliary  tasks  can  actually  interfere  with 
the  pilot’s  evaluation  of  the  characteristics  of  greatest  interest  but,  of 
course,  must  ultimately  be  taken  into  account. 


No  hard  ^d  fast  rule  can  be  given  for  defining  the  evaluation  task 
because  it  is  obvious  that  the  program  objectives  will  play  a  signific^t 
part,  but  the  nearer  the  selected  task  represents  all  the  demands  of  the  real 
mission  during  the  flight  pha:se  or  subphase  being  evaluated,  the  less  will  be 
the  extrapolation  required  of  the  pilot. 

For  pilot  ratings  that  apply  to  a  complete  flight,  flight  phase,  or 
siibphase,  the  rating  must  consider  all  disturbances  specified  for  the  mission 
in  the  sense  of  being  required  operations.  Thus,  more  extrapolation  is 
expected  for  these  ratings,  placing  greater  reliance  on  the  pilot's  judgement 
than  for  either  task  or  composite  ratings.  On  the  other  hand,  task  and  com¬ 
posite-ratings  must  be  reviewed  with  caution,  as  they  may  not  include  the  most 
critical  conditions  and,  therefore,  could  be  misleading. 


The  selection  of  a  complete  flight,  flight  phase,  or  subphase  as  the  unit 
to  be  rated  will  therefore  depend  upon  the  program  objective,  the  character¬ 
istics  of  primary  interest,  and  the  degree  to  which  it  - is  desired  to  draw  upon 
the  pilot's  training,  e:q)erience,  and  knowledge  to  assess  and  extrapolate 
beyond  the  specific  task  provided.  Several  aspects  of  task  selection  involved 
in  actual  practice  are  discussed  further  in  the  following  paragraphs. 

Task  vs.  flight  phase  or  composite  ratings.-  To  provide  a  rating  based  on 
the  simulated  task  is,  of  course,  the  most  direct  approach  and  reduces  the 
uncertainties  of  asking  the  pilot  for  extrapolation.  It  is  weak,  however,  in 
that  the  task  selected  may  not  represent  critical  flight  conditions,  thereby 
allowing  a  nonconservative  leniency  in  the  interpretation  of  the  pilot  rating. 
Obviously,  the  pilot  will  not  be  as  influenced  by  mission  objectives 
told  to  confine  his  rating  to  the  evaluation  task,  although  it  may  be  diffi¬ 
cult  for  him  to  exclude  considerations  that  he  knows  he  would  consider  imder 
actual  operating  conditions.  If,  however,  all  the  separate  t^ks  or  subphases 
needed  to  define  a  flight  phase  can  be  provided,  separate  ratings  may  be 
obtained  for  each  task  or  subphase,  and  a  composite  rating  then  given  for  the 
multiple  tasks  or  subphases  required  to  represent  a  flight  phase.  Providing 
all  the  separate  tasks  required  to  define  a  flight  phase,  however,  is  seldom 
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possible  even  by  simulation.  Nor  is  it  always  possible  in  actual  flight, 
although  its  importance  may  dictate  flying  to  remote  areas  to  find 
environmental  conditions  for  the  actual  tests. 

If  it  is  then  not  possible  to  provide  all  the  separate  tasks  required  to 
define  a  given  flight  phase,  it  may  be  desirable  to  base  a  flight  phase  rating 
largely  on  the  pilot's  ability  to  draw  on  his  knowledge,  experience,  and  use 
of  pilot-induced  disturbances  or  self-induced  tasks.  Such  a  procedure 
utilizes  the  maximum  capability  of  the  pilot  but  also  introduces  extrapolation, 
with  the  inherent  risk  of  his  occasionally  overlooking  a  potential  problem  or 
of  being  unduly  critical  of  certain  characteristics.  The  extent  to  which  the 
pilot  is  expected  to  compensate  for- limitations  in  the  simulated  situation 
must  be  clearly  understood  at  the  start  of  a  program. 

If  the  time  factor  and  probability  of  occurrence  are  the  same  for  each 
task,  the  composite  rating  will  likely  reflect  the  most  critical  task.  If  the 
probability  of  occurrence  is  low  for  one  of  the  tasks  (as  might  be  attribut¬ 
able  to  severe  turbulence),  it  is  still  possible  for  the  probability  of 
encounter  to  influence  a  composite  rating  in  an  undefined  manner  unless  such 
a  task  is  excluded  from  the  composite  rating.  The  experimenter  must  decide 
whether  composite  ratings,  which  include  "probability  of  failure"  or  probabil¬ 
ity  of  encountering  "unusual"  environmental  conditions,  are  to  be  assessed  by 
the  pilot  or  someone  else.  There  must  be  no  doubt  as  to  whose  responsibility 
this  is,  however. 

Transient  disturbances.-  When  a  steady-state  failure  is  designated,  the 
pilot  is  exempt  from  considering  the  probability  of  a  failure.  However,  some¬ 
one  must  consider  the  probability  of  encountering  transient  disturbances  from 
engine  or  other  failures,  wind  shear  or  other  short-term  environmental  condi¬ 
tions.  In  these  cases,  the  pilot  might  have  to  consider  the  duration  of  the 
disturbance  as  well  as  its  level.  If  specified  in  the  mission  definition,  the 
pilot  must  consider  disturbances  whether  they  are  provided  in  the  task  or  not. 

Admittedly,  a  rating  that  reflects  extrapolation  for  the  effect  of  an 
undefined  level  and  extent  of  a  disturbance  represents  an  approximation,  but 
this  may  often  be  preferable  to  ignoring  the  disturbance. 

The  effect  of  certain  disturbances,  such  as  those  induced  by  the 
environment  and  the  dynamics  of  failure,  on  pilot  rating  will  depend  heavily 
upon  the  aircraft  state.  As  a  rule,  transient  disturbances  from  whatever 
source  will  affect  the  pilot  rating  less  if  the  handling  qualities  are  good 
rather  than  poor.  For  this  reason,  it  becomes  increasingly  important  to  pro¬ 
vide  more  accurate  dynamic  representation  of  transient  disturbances  as 
handling  qualities  deteriorate. 

Single  axis  or  single  parameter  evaluations.-  Confusion  can  occur  when 
ratings  are  requested  for  specific  parameters  or  for  a  single  axis  rather  than 
for  the  complete  pilot-vehicle  combination  in  performance  of  its  mission.  It 
should  be  hcmdiing  qualities  that  are  rated,  with  the  influence  of  specific 
parameters  identified  through  changes,  if  any,  in  the  pilot  ratings  and  from 
the  pilot  comment  data. 
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Single-axis  simulations  cam  be  of  considerable  value  in  quick  look 
evaluations  but  should  be  clearly  identified,  so  that  the  limitations  of  the 
data  can  be  recognized.  The  taisk  that,  for  example,  includes  freedom  for 
mameuvering  in  pitch  only  cannot  provide  information  on  either  total  pilot 
workload  or  control  coupling.  When  a  single  axis  or  single  parameter  is  of 
primary  interest,  it  is  preferable  to  incorporate  the  single  axis  characteris¬ 
tics  into  a  full  multiple-axis  (six  degrees  of  freedom)  task  so  that  the 
evaluation  can  be  made  in  the  full  context  of  handling  qualities. 

Deficiencies  evaluated  in  the  longitudinal  flight  mode  cannot  be  sin^jly 
added  to  deficiencies  evaluated  separately  in  the  lateral -directional  mode. 

As  a  consequence,  the  most  critical,  single,  axis. .task,  .cannot  .necessarily  be 
assumed  to  be  the  most  critical  overall  pilot-vehicle  task. 

A  parallel  situation  arises  in  the  general  application  of  quantitative 
criteria  for  handling  qualities.  As  a  rule,  such  criteria  are  developed  for 
specific  parameters  with  all  other  parameters  at  a  satisfactory  level.  Such 
is  not  necessarily  the  case  in  real  life,  so  that  one  cannot  expect  to  find 
single  parameter  criteria  that  apply  accurately  to  all  aircraft  with  their 
various  roles,  configurations,  and  operating  environments. 

Preoccupation  with  variations  in  a  single  parameter  should  not  be  allowed 
to  distract  the  pilot  from  the  fact  that  his  ratings  apply  to  handling  quali¬ 
ties  as  a  whole,  for  the  task  (or  flight  phase)  in  context  of  the  aircraft 
role.  Assessment  of  single  parameters  or  single  axis  effects  can  be  very 
helpful  to  a  designer,  but  special  care  must  be  taken  in  the  development  and 
presentation  of  such  data.  Programs  of  such  limited  scope  are  likely  to 
introduce  considerable  pilot  extrapolation  with  a  resulting  low  level  of  con¬ 
fidence  so  that  in  some  cases  it  may  be  more  desirable  to  obtain  only  pilot 
comment  data  or  comparative  results  not  associated  with  a  handling  qualities 
rating. 

An  evaluation  may  often  involve  comparisons  or  relative  assessment  of 
specific  characteristics,  but  the  evaluation  pilot  should  not  become  so 
engrossed  in  comparisons  that  he  loses  sight  of  the  absolute  aspects  of  per¬ 
formance  and  workload  in  the  context  of  the  role  and  mission  requirements. 

The  xise  of  an  entirely  separate  rating  scale,  specifically  for  relative  assess¬ 
ment  of  individual  stability  and  control  parameters,  has  been  considered  but 
abandoned  because  of  possible  confusion  introduced  by  a  second  scale. 


Pilot  Assessment  Considerations 

Pilot  rating.-  As  a  shorthand  representation  of  the  handling  qualities  of 
an  aircraft  in  the  performance  of  a  defined  mission  and  task,  a  pilot  rating 
will  be  meaningful  only  in  proportion  to  the  care  taken  in  developing  the  pro¬ 
gram  (defining  objectives,  the  role  and  mission,  the  evaluation  task,  what  the 
rating  applies  to,  the  simulation  situation  and  extent  of  pilot  extrapolation 
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involved) .  Unless  a  common  basis  is  established  and  any  criteria  used  are 
clearly  indicated,  one  cannot  expect  to  achieve  reliable  data  and  comparable 
ratings  among  pilots.  Large  disagreement  between  pilot  ratings  is  usually 
traced  to  incomplete  program  development. 

There  tends  to  be  some  disagreement  among  pilots  as  to  how  they  actually 
arrive  at  a  specific  numerical  rating.  Some  pilots  lean  heavily  on  the  speci¬ 
fic  rating  description  and  look  for  the  description  that  best  fits  their  over¬ 
all  assessment.  Other  pilots  prefer  to  make  the  dichotomous  decisions 
sequentially,  thereby  arriving  at  a  choice  between  two  or  three  ratings.  The 
decision  among  the  two  or  three  ratings  is  then  based  upon  the  adjective 
description.  In  concept,.,  the.  latter  -technique_is.  preferable,  since.it  empha¬ 
sizes  the  relationship  of  all  quality  decisions  to  the  aircraft  role  and 
mission  requirements.  With  the  final  version  of  the  revised  scale,  the  pilot 
decisions  shown  must  be  considered  in  order  to  define  the  category  boundaries. 

The  actual  technique  used  is  probably  somewhere  between  the  two  tech¬ 
niques  discussed  above  and  is  not  so  different  among  pilots.  In  the  past,  the 
pilot's  choice  has  probably  been  strongly  influenced  by  the  relative  useful¬ 
ness  of  the  descriptions  provided  for  the  categories  on  one  hand,  and  the 
numerical  ratings  on  the  other.  The  evaluation  pilot  is  more  or  less  continu¬ 
ously  considering  the  rating  decision  process  during  his  evaluation.  He  pro¬ 
ceeds  through  the  dichotomous  decisions  to  the  adjective  descriptors  enough 
times  that  his  final  decision  is  a  blend  of  both  techniques. 

Half  ratings  (e.g.,  rating  4-1/2)  generally  indicate  reluctance  of  the 
evaluation  pilot  to  assign  either  of  the  adjacent  ratings  to  describe  the  con¬ 
figuration.  Any  finer  breakdown  than  half  ratings  is  hardly  ever  justified 
since  ^y  number  greater  than  or  less  than  the  half  rating  iii5)lies  that  it 
belongs  in  the  adjacent  group.  Any  distinction  between  configurations 
assigned  the  same  rating  must  be  made  in  the  pilot  comments.  Use  of  the 
3-1/2,  6-1/2,  and  9-1/2  ratings  is  discouraged  because  they  represent  impor¬ 
tant  "boundary"  conditions  and  the  decision  as  to  which  category  is  selected 
may  be  significant  even  though  hard  to  make. 

As  a  general  rule,  pilot  ratings  and  comments  are  preferably  given  on  the 
spot  when  the  characteristics,  performance,  and  workload  are  fresh  in  mind. 

If  the  pilot  should  later  want  to  change  his  rating,  the  reasons  for  the 
change  may  be  of  interest.  In  some  cases,  an  attempt  should  be  made  to  repeat 
the  configuration  later  in  the  evaluation  program. 

Pilots'  comments.-  The  use  of  a  rating  scale  considered  for  universal 
handling  qualities  application  leads  to  the  assumption  that  the  numerical 
pilot  rating  can  represent  the  entire  qualitative  assessment.  Extreme  care 
must  be  taken  against  this  oversimplification  because  the  numerical  ratings  do 
not  constitute  the  complete  results  of  the  data  gathering  process. 

As  one  might  expect,  the  evaluation  data  most  often  neglected  are  the 
pilots'  comments,  either  because  they  are  not  recorded  or  because  they  are 
often  difficult  to  deal  with  because  of  their  qualitative  form,  and  perhaps 
their  bulk.  Often  the  nature  of  the  experiment  allows  the  pilot  neither  the 


23 


time  nor  the  opportvinity  to  identify  the  cause  of  his  objections.  Ratings, 
without  the  pilot's  objections,  are  only  part  of  the  story.  Only  if  the  defi¬ 
cient  areas  can  be  identified,  can  one  expect  to  devise  improvements  to  elimi¬ 
nate  or  attenuate  the  shortcomings .  The  pilot  comments  are  the  means  by  which 
the  identification  can  be  made. 

The  pilots'  objections  to  the  handling  qualities  are  inqjortant  therefore 
to  the  airplane  designer  who  is  responsible  for  in5)roving  the  handling  qual¬ 
ities  and  to  the  engineer  who  is  attempting  to  understand  and  use  the  pilot 
rating  data.  If  ratings  are  the  only  output,  the  engineer  has  no  real  way  of 
assessing  whether  the  objectives  of  the  experiment  were  actually  realiz^.  A 
pilot's  comments  supply  a  means  of  assessing  whether  his  objections  (which 
lead  to  his  summary  rating)  were  related  to  the  mission,  to  some  extraneous 
factor  in  the  execution  of  the  experiment,  or  to  his  inaccurate  interpretation 
of  vari<His  aspects  of  the  mission.  For  pilots'  comments  to  be  most  useful, 
several  details  must  be  kept  in  mind. 

Generally,  pilots  should  comment  in  the  simplest  possible  language. 
Attempts  to  translate  the  observed  characteristics  or  responses  into  engineer¬ 
ing  terms  should  be  avoided,  unless  such  terms  accurately  supplement  the 
pilot's  observations  given  in  descriptive  terms.  The  pilot  should  report  what 
he  sees  and  feels,  and  describe  his  difficulties  in  carrying  out  whatever  he 
is  attempting.  It  is  then  important  for  the  pilot  to  relate  his  difficulties 
in  executing  specific  tasks  to  their  effect  on  the  accomplishment  of  the 
required  operations. 

The  pilot  should  be  encouraged  to  make  specific  comments  when  evaluating 
each  configuration.  These  comments  generally  are  in  response  to  questions 
develop^  during  discussions  of  the  mission  and  simulation  situation.  The 
pilot  must  also  be  encouraged  to  comment  regarding  his  difficulties  over  and 
above  the  answers  to  the  specific  questions  asked  of  him.  In  this  regard,  the 
test  pilot  should  strive  for  a  balance  between  a  continuous  running  commentary 
and  only  occasional  comment  in  the  form  of  an  explicit  adjective.  The  former 
often  requires  so  much  editing  to  find  the  substance  that  it  is  often  ignored, 
while  the  latter  may  add  nothing  to  the  numerical  rating  itself. 

The  pilot's  comments  must  be  collected  during  or  immediately  after  each 
evaluation.  If  the  conments  are  left  until  the  conclusion  of  the  evaluation, 
they  are  often  forgotten.  For  both  in-flight  and  ground  simulator  evaluations, 
this  means  that  provision  should  be  made  for  wire  or  tape  recording.  The 
immediate  recording  of  key  words  or  phrases  on  a  kneepad  can  often  be  easily 
expanded  to  the  full  content  after  the  task  or  evaluation  is  completed.  It 
has  been  the  ejqjerience  of  the  authors  that  the  best  voluntary  comments  are 
often  given  during  the  evaluation.  A  useful  procedure  is  to  encourage  volun¬ 
tary  comment  and  note-taking  during  the  evaluation  and  to  require  answers  to 
specific  questions  in  the  summary  comments  at  the  end  of  the  evaluation. 

Questionnaires  or  explicit  check  lists  ensure  that:  (a)  all  iiiportant  or 
suspected  aspects  are  considered  and  not  overlooked,  (b)  the  reason  for  a 
given  rating  is  specified,  (c)  an  understanding  is  provided  of  the  tradeoffs 
with  which  pilots  must  continually  contend,  and  (d)  supplementary  comment  that 
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1 


Description 

No  tendency  for  pilot  to  induce  undesiroble  motions. 


Numeral 

I 


Undesirable  motions  tend  to  occur  when  pilot  initiotes  obrupt  moneuvers 
or  attempts  tight  control.  These  motions  can  be  prevented  or  eliminated  2 

by  pilot  techniques. 


Undesirable  motions  easily  induced  when  pilot  initiotes  abrupt  moneuvers 
or  attempts  tight  control.  These  motions  can  be  prevented  or  eliminated  3 
but  only  at  sacrifice  to  task  performonce  or  through  considerable  pitot 
ottention  and  effort. 


Oscillations  tend  to  develop  when  pilot  initiotes  obrupt  maneuvers  or 
ottempts  tight  control.  Pilot  must  reduce  gom  or  abandon  tosk  to  recover. 

Divergent  oscillotions  tend  to  develop  when  pilot  initiotes  abrupt  moneuvers 
or  attempts  tight  control.  Pilot  must  open  loop  by  releasing  or  5 

freezing  the  stick. 

Disturbance  or  normal  pilot  control  moy  cause  divergent  oscillation.  Pilot  ^ 
must  open  control  loop  by  releasing  or  freezing  the  stick. 


♦  These  numerals  are  not  reloted  to  the  pitot  rotmg  scale. 


Figure  8.-  Classification  of  PIO  tendency. 


might  not  be  offered  otherwise  is 
stimulated.  It  is  recommended  that 
the  pilots  participate  in  the  prepar¬ 
ation  of  the  questionnaires.  The 
questionnaires  should  be  modified  if 
necessary  as  a  result  of  the  pilots’ 
initial  evaluations .  On  occasion,  it 
may  be  desirable  to  classify  pilots' 
comments  by  having  the  pilots  select 
one  of  several  ranked  comments  about 
a  specific  characteristic.  An  exam¬ 
ple  of  such  a  classification  of  spe¬ 
cific  pilots'  comments  is  shown  in 
figure  8  for  PIO  tendency.  Classifi¬ 
cation  as  shown  in  this  example  is 
for  easy  identification  only,  and  is 
not  to  be  confused  with  the  designa¬ 
tion  for  pilot  rating. 


Simulation  Situation 

A  pilot  evaluation  is  seldom  conducted  under  the  full  circumstances  of  an 
actual  flight.  It  almost  always  involves  simulation  to  some  degree  because  of 
the  absence  of  the  real  situation.  As  an  example,  the  evaluation  of  a  day 
fighter  is  seldom  carried  out  under  the  circumstances  of  a  combat  mission 
where  the  pilot  is  not  only  shooting  at  real  targets  but  is  being  shot  at  with 
real  bullets.  Therefore,  after  the  program  has  been  defined  and  the  decision 
made  as  to  what  is  being  rated,  the  relationship  of  the  simulation  to  the  real 
situation  must  be  explicitly  stated  for  both  the  engineer  and  the  evaluation 
pilot  SO'  that  each  may  clearly  understand  the  limitations  with  respect  to  what 
is  provided  in  the  task,  the  environmental  disturbances,  the  cockpit  interface, 
and  the  completeness  of  the  simulation  of  the  stability  and  control 
characteristics . 

The  pilot  and  engineer  must  know  not  only  what  is  left  out  of  an 
evaluation  program,  but  also  what  is  in  that  should  not  be  in.  The  fact  that 
the  anxiety  and  tenseness  of  the  real  situation  are  missing,  and  that  the  air¬ 
plane  is  flying  in  the  clear  blue  of  calm  daylight  air  instead  of  in  the  icy, 
cloudy,  turbulent,  darkness  of  the  real  mission  could  affect  the  results  and 
should  be  considered  during  the  program  development  and  the  interpretation  of 
the  results . 

Cockpit  interface.-  The  cockpit  interface  must  be  specified,  at  least  to 
the  extent  necessary  to  identify  any  factors  that  influence  handling  quali¬ 
ties.  As  a  rule,  once  this  is  done  for  a  given  program  it  remains  fixed  and 
is  merely  reference  information.  IVhen  stability  and  control  characteristics, 
for  example,  are  the  primary  interest,  features  pertaining  to  the  flow  of 
information  to  and  from  the  cockpit  probably  would  not  be  considered  as  vari¬ 
ables  in  the  study,  but  they  must  be  noted  with  sufficient  detail  to  establish 
at  least  any  possible  qualitative  influence  on  the  results. 
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The  individual  cockpit  features,  such  as  control  system  characteristics 
and  instrument  display,  may,  however,  often  include  the  specific  variables  to 
be  studied.  In  some  cases,  these  variables  will  be  made  known  to  the  evalua¬ 
tion  pilot,  while  in  others,  it  will  be  more  important  that  he  assess  only  the 
overall  handling  qualities  with  respect  to  task  accon^lishment . 

Unless  control  system  characteristics  are  a  variable  to  be  studied,  it  is 
desirable  to  provide  an  accurate  representation  of  the  anticipated  vehicle 
control  system.  Such  factors  as  control  system  break-out  force,  friction, 
inertial  characteristics,  or  system  gearing  and  sensitivity  may  significantly 
affect  pilot  rating.  If  it  is  not  practical  to  represent  the  vehicle  control 
system  characteristics  accurately  then  system  characteristics  should  be  pro¬ 
vided  that  are  at  least  satisfactory  to  the  evaluation  pilots  and  do  not 
introduce  unwanted  problems.  It  is  often  desirable  to  allow  evaluation  pilots 
to  establish  comfortable  levels  of  control  sensitivity,  for  example,  before 
beginning  an  evaluation  program.  This  procedure  has  distinct  advantages  in 
reducing  the  number  of  uncontrolled  variables  that  may  enter  into  an  evalua¬ 
tion  program.  Care  must  be  taken,  however,  that  this  procedure  is  not  used 
when  the  control  sensitivity  is  a  variable. 

Repositioning  any  of  the  selectors  available  to  the  flight  crew  to  change 
aircraft  configuration  will  depend  on  the  flight  phase  or  task  being  evaluated. 
Visual  information  available  to  the  pilot  is  defined  by  such  features  as  the 
cockpit  cut-off  angle,  the  simulator  system  for  providing  external  information, 
and  the  types  and  locations  of  the  cockpit  instruments. 

Aircraft  environment.-  The  extent  to  which  aircraft  environmental  factors 
are  included  in  the  simulation  must  be  described.  It  is  important  that  the 
level  of  such  disturbances  be  related  to  the  length  of  time  they  are  to  be 
coped  with.  This  is  usually  for  the  duration  of  a  flight  phase,  or  specified 
task,  but  if  it  is  to  be  considered  as  a  transient  condition,  this  should  be 
noted . 

Should  the  evaluation  be  conducted  under  actual  flight  conditions,  it  may 
be  desirable  to  secure  a  turbulence  classification  based  on  the  standardized 
scale  from  the  FAA  Airman's  Information  Manual  (fig.  9),  or  to  relate  to  other 
documentation  as  may  be  available. 

The  probability  of  occurrence  of  various  levels  of  turbulence  is  an 
iii5)ortant  consideration,  but  is  much  too  complex  for  the  pilot  to  consider 
during  an  evaluation. 

The  influence  of  environmental  disturbances  varies  inversely  with  the 
level  of  handling  qualities;  that  is,  in  general,  the  poorer  the  handling 
qualities,  the  more  they  are  influenced  by  adverse  disturbances.  There  is  no 
simple  rule,  however,  for  translating  ratings  obtained  in  smooth  air  to 
ratings  applicable  to  rough  air. 

Environmental  disturbances  with  some  undefined  probability  of  occurrence 
can  strongly  influence  the  aircraft  design  and  its  handling  qualities.  For 
th-is  reason,  their  inclusion  can  be  essential  to  the  conduct  of  handling  qual¬ 
ities  experiments.  When  no  provision  is  made  for  actual  or  simulated 


26 


'^Intensity 

Aircraft  reaction 

Reaction  inside  aircraft 

Reporting  term 

Definition  ^ 

LIGHT 

Turbulence  that  momentorily  causes  slight  chonges  in  oircroft 
attitude,  altitude,  or  heodinq.  Report  as  Light  Turbulence: 

OR 

Turbulence  (light  bumpiness)  thot  causes  slight  oircroft 
fluctuotions  at  rapid  intervols  without  opprecioble  change  in 
altitude,  roll,  or  yow.  Report  os  Light  Chop. 

Occuponts  may  feel  o  slight  stroin  against 
sect  belts  or  shoulder  strops. 

Unsecured  objects  remom  at  rest,  food 
service  moy  be  conducted,  and  little  or 
no  difficulty  is  encountered  in  walking. 

Occasional 

Intermittent 

Continuous 

Less  then  '/j 
of  the  time 

to  % 

More  than  2/3 

MODERATE 

Turbulence  that  is.similor  to  Light  Turbulence  but  of  greater 
intensity.  Chonges  in  oircroft  attitude,  oltitude,  or  heading 
occur,  but  the  oircroft  remains  in  positive  control  at  all  times. 
Report  as  Moderote  Turbulence. 

OR 

Turbulence  that  is  similor  to  Light  Chop  but  of  greeter 
intensity.  It  couses  rapid  bumps  or  jolts  without  opprecioble 
change  m  oircroft  altitude,  roll,  or  yow. 

Report  os  Moderate  Chop. 

Occupants  feel  definite  stroins  ogoinst 
seat  belts  or  shoulder  strops. 

Unsecured  objects  ore  dislodged. 

Food  service  and  wolking  ore  difficult. 

NOTE 

1.  Pilots  should  report  location(s), 
time,  intensity,  altitude,  type  of 
oircroft  ond,  when  applicable, 
duration  of  turbulence. 

2.  Duration  moy  be  based  on  time 
between  two  locations  or  over  a 
single  location.  All  locations  should 
be  readily  identifioble, 

EXAMPLES 

a.  Over  Omaho,  1232Z,  moderate 
turbulence,  Flight  Level  300, 
B707 

b.  From  50  miles  south  of 

Albuquerque  to  30  miles  north  of 
Phoenix,  1235Z  to  1250Z, 
occasional  Moderate  Chop,  Flight 
Level  320,  DC* 8 

_ J 

SEVERE 

Turbulence  thot  couses  large  chonges  m  aircraft  oltitude, 
attitude,  or  heading.  It  moy  couse  large  voriotions  in  indicated 
oirspeed.  Aircraft  may  be  momentarily  out  of  control 

Reoort  as  Severe  Turbulence. 

Occuponts  are  forced  violently  and 
repeatedly  ogainst  seat  belts  or  shoulder 
straps. 

Unsecured  objects  are  tossed  obout. 

Food  service  and  wolking  are  impossible. 

EXTREME 

V 

Turbulence  in  which  the  oircroft  is  violently  tossed  about  and 
is  practically  impossible  to  control.  It  moy  cause  structuro! 
damage.  Report  as  Extreme  Turbulence. 

Figure  9.-  Turbulence  criteria. 


environmental  disturbances,  it  may  be  necessary  for  the  pilot  to  learn  their 
effect  on  handling  qualities  by  employing  self-imposed  disturbances.  Such 
procedures  have  limited  application  in  a  sophisticated  simulation  program,  but 
do  have  considerable  value  in  "short  look"  programs. 

Pilot  stress.-  Surprise  and  stress  can  interfere  with  the  pilot's 
performance  by  dFstracting  him  from  the  primary  task.  Considering  these  fac¬ 
tors  in  depth  is  usually  left  to  the  more  sophisticated  programs,  but  those 
elements  that  contribute  to  greater  fidelity  and  realism  in  a  simulation  pro¬ 
gram  will  be  found  desirable.  The  sharp  jolt  of  the  cockpit  when  a  bad  land¬ 
ing  occurs  contrasts  vividly  with  the  smooth  conditions  of  a  good  landing. 

An  element  of  surprise  and  stress  should  be  introduced  when  the  dynamic 
effects  of  control  system  or  SAS  failures  are  being  evaluated.  To  be  truly 
objective  would  require  data  from  many  occurrences  introduced  in  a  random  man¬ 
ner  during  the  time  the  pilot  is  engrossed  in  the  performance  of  a  task. 

If  surprise  and  stress  are  not  included  in  the  task,  the  evaluation  pilot 
must  simulate  surprise,  distraction,  or  unattended  operation  by  using  a 
delayed  reaction  time.  The  stress  or  rapid  increase  in  pilot  workload  when 
two  or  more  adverse  conditions  or  situations  occur  together  can  be  studied  in 
more  sophisticated  simulation  programs.  These  potential  "operating  problems" 
consisting  of  combinations  of  failures  or  weather  conditions  are  apt  to  raise 
the  question  of  probability  of  occurrence.  Again,  this  question  should  be 
separated  from  the  pilot  assessment  whenever  possible.  The  pilot  rating 
should  reflect  the  difficulty  and  effect  on  performance  of  the  occurrence 
relative  to  successful  completion  of  the  task  or  flight  phase  under  evaluation. 
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Therefore,  such  occurrences  must  originate  with  the  program  objective  and  be 
documented  in  the  test  plan  as  to  how  they  will  be  executed;  that  is,  whether 
they  will  be  included  in  a  task  or  left  to  the  pilot's  individual  assessment. 

Pilot  extrapolation  and  confidence.-  The  question  of  simulation  enters 
into  nearly  every  evaluation  program  to  some  degree.  Previous  studies 
(ref.  9)  have  shown  that  sophistication  is  not  necessarily  the  key  to  simula¬ 
tor  usefulness  although  it  can  extend  the  range  of  application.  Deciding 
"what  a  pilot  rating  applies  to"  (specific  task  or  flight  .phase) ,  and  the 
completeness  of  the  simulation  will  determine  the  degree  to  which  pilot 
extrapolation  is  to  be  relied  on.  Neither  the  pilot  nor  the  engineer  retains 
confidence  in  the  results  if  the  need  for  extrapolation  of  observed  results 
becomes  too  great.  Guidelines  have  been  drawn  in  this  report  to  guard  against 
this  problem  while  at  the  same  time  encouraging  full  utilization  of  the 
trained  and  experienced  test  pilot. 


It  is  felt  that  careful  planning  and  agreement  on  program  objectives, 
mission  definition,  what  is  being  rated,  and  the  execution  of  the  experiment 
can  limit  the  uncertainties  of  extrapolation. 

In  order  to  provide  a  means  for  expression,  on  the  part  of  pilots  in 
particular,  a  confidence  factor  could  be  introduced  into  programs  in  which 
simulation  is  involved.  A  definition  might  be  "the  ratio  of  the  information 
available  (to  the  pilot)  in  the  simulation  situation  to  the  information 
required  to  derive  a  realistic  pilot  rating."  While  provision  is  made  for  the 
use  of  such  a  confidence  factor,  no  recommendation  as  to  its  use  is  given  in 
this  report.  If  used,  care  must  be  taken  to  insure  that  the  confidence  factor 
does  not  inhibit  or  otherwise  confuse  the  actual  pilot  ratings  assigned.  As 
reference  information,  a  confidence  factor  could  help  a  pilot  focus  his  com¬ 
ments  on  the  adequacy  of  the  simulation,  but  questionnaires  could  serve  the 
s  ame  purpos  e . 


•  CLASS  A 

A  pilot  may  usually  assign  o  rating  with  a  reiotively  high  degree 
of  confidence,  olthough  be  may  hove  mild  reservations  because  of 
incomplete  or  inadequate  simulotion  of  motion  cues,  disturbances, visual 
information,  or  other  factors  effecting  pilot  worklood. 

Supplementary  tasks,  if  needed,  can  be  odequately  provided  by 
the  pilot. 

•  CLASS  B 

A  pilot  can  assign  a  rating  with  only  a  moderote  level  of 
confidence  because  of  uncertainties  introduced  by  a  lack  of  representative 
environmental  disturbances  os  well  as  incomplete  or  inadequate  simulation 
of  motion  cues,  disturbances,  visuoi  information,  or  other  factors 
affecting  pilot  workload. 

Supplementary  tosks  moy  be  desired,  but  are  not  ovailoble. 

•‘class  C 

A  pilot  can  assign  a  rating  with  only  minimum  confidence  because 
considerable  pilot  extrapolation  is  required  due  to  an  incomplete  task, 
thereby  requiring  considerable  reliance  on  self-imposed  tosks  ond 
maneuvers  for  assessment. 

This  moy  also  be  aggravated  by  incomplete  or  very  limited 
simulotions  of  motion  cues,  disturbances,  visual  information,  or 
other  factors  affecting  pilot  workload. 


For  those  who  favor  using  a 
confidence  factor,  a  simple  classi¬ 
fication  is  proposed  in  figure  10, 
as  A,  B,  or  C.  Studies  involving 
"quick  look"  or  rudimentary  simula¬ 
tion  may  often  call  for  pilot  rat¬ 
ings  in  which  the  pilot  is  confident 
in  only  a  relative  rather  than  an 
absolute  sense.  For  these,  a  "C" 
classification  might  be  appropriate. 
An  attempt  should  be  made  to  place 
all  other  evaluations  in  Class  A, 
thereby  allowing  Class  B  to  be  used 
only  when  a  pilot  has  difficulty 
deciding  between  A  and  C. 


Figure  10.-  Classification  of  pilot  confidence  factor. 
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The  Briefing  Guide 


In  order  to  summarize  the  content  of  this  report  in  a  form  which  may  be 
applied  directly  to  the  execution  of  handling  qualities  experiments,  a 
"Briefing  Guide"  has  been  assembled  in  appendix  B.  The  purpose  of  this  guide 
is  to  outline  a  format  to  insure  that  all  pertinent  documentation  is  covered 
for  each  evaluation  program.  Only  in  this  way  can  the  evaluation  pilots  be 
sure  that  everyone  is  talking  about  the  same  thing.  Subsequent  analysis  of 
the  data,  reporting,  and  ultimate  conparison  with  data  from  other  sources  is 
then  materially  aided. 

The  briefing  guide  enables  the  pilot  and  engineer  to  know  what  is  missing 
from  an  evaluation  program,  as  well  as  what  is  provided. 

Once  the  information  is  tabulated,  it  should  be  apparent  just  what  is 
being  evaluated  and  rated,  what  the  task  is,  the  conditions  under  which  the 
evaluation  is  performed,  and  the  applicability  of  the  data.  With  the  impor¬ 
tant  information  provided  in  a  Briefing  Guide,  it  should  be  possible  to 
evaluate  handling  qualities  more  effectively  and  to  improve  communication 
not  only  between  pilots  and  engineers,  but  between  various  research  and 
development  groups  as  well. 


Ames  Research  Center 

National  Aeronautics  and  Space  Administration 

Moffett  Field,  Calif.  94035,  Nov.  13,  1968 
125-19-06-06-00-21 
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APPENDIX  A 


COMPENSATION  - 

CONFIGURATION  - 


COCKPIT  INTERFACE  - 


CONTROLS  - 


Primary  - 


Secondary  - 


Selectors  - 


DEFINITIONS 


The  measure  of  additional  pilot  effort  and  attention 
required  to  maintain  a  given  level  of  performance  in  the 
face  of  deficient  vehicle  characteristics. 

The  aircraft  geometry  as  established  by  the  actual 
position  of  movable  portions  and  surfaces  controllable 
by  the  selectors,  and  the  state'  of  operability  of 
on-board  systems. 

Exan5)les:  Auto- throttle  on,  flaps  at  15®,  wing  sweep 
at  45®,  etc. 

The  means  provided  for  the  flow  of  information  to  and 
from  the  pilot.  These  include  the  display  of  informa¬ 
tion  available  to  the  pilot  as  well  as  the  type  and 
characteristics  of  the  cockpit  controls . 

Exan^les:  Description  of  cockpit  instruments  and  layout. 

Description  of  control  system  (i.e.,  stick, 
wheel,  force  -  deflection  gradients,  sensi¬ 
tivities,  primary,  secondary  controls,  and 
selectors  provided) . 

A  distinction  is  made  between  the  types  of  controls  in 
the  cockpit  according  to  their  function.  Principal  con¬ 
trols  are  the  primary  and  secondary  controls . 

Those  controls  used  by  a  pilot  to  continuously  modify 
the  movement  of  the  aircraft. 

Examples:  Pitdi,  roll,  yaw  controls,  throttle,  DLC. 

Those  controls  used  by  a  pilot  to  make  discrete  changes 
in  the  movement  or  balance  of  the  aircraft,  thereby  modi¬ 
fying  the  need  for  actuation  of  the  primary  controls . 

Examples:  Pitch,  roll,  and  yaw  trimmers,  aerodynamic 
braking  devices. 

Those  cockpit  controls  available  to  the  crew  for 
changing  aircraft  configuration. 

Exan^les:  Flaps,  slats,  wing  sweep,  BLC. 
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FAILURE  STATE 


FLIGHT  or  SORTIE  - 


FLIGHT  PHASE  - 


FLIGHT  SUBPHASE  - 


HANDLING  QUALITIES  - 


A  steady-state  failure  characterized  by  the  various 
failed  systems  that  affect  the  handling  qualities.  The 
dynamic  effect  of  a  failure  is  called  a  change  of  state 
and  should  be  noted  separately. 

Examples:  Any  failure  resulting  in  loss  of  selected 

function.  Engine  failure,  augmentation  sys¬ 
tem,  failure  in  stability,  autothrottle,  pri¬ 
mary  flight  control  system  (power  boost, 
electric  stick,  servo  control  feel,  etc.)  or 
secondary  flight  control  system  (trim,  aero¬ 
dynamic  brake,  etc.). 

A  complete  sequence  of  flight  phases  of  an  aircraft 
within  one  of  its  roles.  Full  or  con5)lete  mission. 

Example:  The  composite  of  takeoff,  climb,  cruise,  com¬ 

bat  (or  other  special  phase),  descent, 
approach,  landing. 

A  designated  portion  or  segment  of  a  complete  flight.  A 
mission  phase.  A  flight  phase  may  be  represented  by  one 
or  more  separate  tasks. 

Examples:  (a)  Common  phases  -  takeoff,  climb,  cruise, 

descent,  approach,  and  landing. 

(b)  Special  phases  required  by  role  - 
formation,  refueling,  air-to-air  or  air-to- 
ground  combat,  weapon  delivery,  emergency 
conditions  (i.e.,  2-  or  3-engine  operation, 
emergency  descent,  etc.),  VTOL  transition, 
VTOL  hover,  STOL  takeoff,  and  STOL  approach. 

That  part  of  a  flight  phase  having  a  single  objective, 
and  a  single  configuration  or  change  in  a  configuration. 

Examples:  Air-to-air  tracking,  terminal  area  holding, 
glide  slope  capture,  localizer  capture,  ILS 
tracking,  wave-off. 

Those  qualities  or  characteristics  of  an  aircraft  that 
govern  the  ease  and  precision  with  which  a  pilot  is  able 
to  perform  the  tasks  required  in  support  of  an  aircraft 
role. 


MANEUVER  - 

MISSION  - 

PERFORMANCE  - 

ROLE  - 

SPECIAL  CONDITIONS 

STATE  - 

TASK  - 

Control  - 


A  planned  and  regulated  movement  of  ein  aircraft  for  the 
purpose  of  aiding  the  completion  of  a  given  control  task 

Examples:  Bank,  turn,  dive,  pullup,  turn  reversal,  roll 
reversal,  rolling  pullup,  steady  sideslip, 
return  from  sideslip,  control  steps  and 
pulses,  maintenance  of  a  steady  condition. 

The  composite  of  pilot-vehicle  functions  that  must  be 
performed  to  fulfill  operational  requirements .  May  be 
specified  for  a  role,  complete  flight,  flight  phase,  or 
flight  subphas e . 

The  precision  of  control  with  respect  to  aircraft  move¬ 
ment  that  a  pilot  is  able  to  achieve  in  performing  a 
task.  (Pilot-vehicle  performance  is  a  measure  of 
handling  performance.  Pilot  performance  is  a  measure 
of  the  manner  or  efficiency  with  which  a  pilot  moves 
the  principal  controls  in  performing  a  task.) 

The  function  or  purpose  that  defines  the  primary  use  of 
an  aircraft. 

The  special  circumstances  pertinent  to  the  evaluation 
(i.e.,  aircraft  environment  eind  pilot  stress). 

Examples:  Special  conditions  of  weather  and  environment 
turbulence,  wind  shear,  ceiling,  visibility  - 
night,  etc.  Pilot  awareness,  surprise,  or 
distraction  with  respect  to  impending  failure 
or  disturbances . 

The  mass  distribution  and  failure  situation  that  deter¬ 
mine  completely  the  behavior  characteristics  of  the 
aircraft.  A  state  without  a  failure  is  a  normal  state. 

The  actual  work  assigned  a  pilot  to  be  performed  in 
completion  of  or  as  representative  of  a  designated 
flight  segment. 

That  part  of  a  tasTc  which  requires  continuing  actuation 
of  the  principal  controls  and  use  of  the  selectors  (see 
••CONTROLS")  as  required. 

Examples:  Movement  between  specified  points,  tracking 
part  of  weapon  delivery,  ILS  or  VOR  tracking. 
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Auxiliary  - 


WORKLOAD  - 


Physical  - 


Mental  - 


That  part  of  a  task  which  involves  the  pilot  in  actions 
other  than  direct  control  of  the  aircraft. 

Examples:  Navigation,  communication  monitoring,  and 
selection  of  systems. 

The  integrated  physical  and  mental  effort  required  to 
perform  a  specified  piloting  task. 

The  effort  expended  by  the  pilot  in  moving  or  imposing 
forces  on  the  controls  during  a  specified  piloting  task. 

Mental  workload  is  at  present  not  amenable  to  quantita¬ 
tive  analysis  by  other  than  pilot  evaluation,  or  indi¬ 
rect  methods  using  physical  workload  (input)  and  the 
task  performance  measurements.  An  example  would  be  the 
improvement  associated  with  flight-director  type  dis¬ 
plays  which  reduce  the  mental  compensation  normally 
required  of  the  pilot. 
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APPENDIX  B 


BRIEFING  GUIDE  AND  RATING  INFORMATION  FOR  HANDLING  QUALITIES  EXPERIMENTS 

PROGRAM  DEFINITION 


Objectives 

General : 

Research  Q  Development  Q  Acceptance  Q  Other  Q 

Aircraft  role _ 

Flight  segment  of  interest _ _ ^ 

Parameter  or  variable  of  primary  interes t  _ 

Mission  description; 

General  statement  of  the  required  operations  for  the  flight  segment 
of  interest  in  context  of  aircraft  role. 

1.  What  is  pilot-vehicle  combination  required  to  accomplish? 


2.  What  are  the  conditions  under  which  these  required  operations  are 
to  be  carried  out  (i.e.,  aircraft  state,  environment,  and  cockpit 
interface) ? 
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Scope 


Short  look  (guide  to  further  tests) Q 
Long  look  (definitive  as  possible) Q 
Task  related  only  (minimize  pilot  extrapolation) fl 
Mission  or  flight  segment  related  (pilot  extrapolation  encouraged) Q 
Nature  of  task  or  tasks  provided:  fl 
Short  term  I  I  Long  term. PI 

Critical  task(s)  included?  1""!  Or  left  for  pilot  extrapolation?  fl 
Identify:  _ _ • 


Are  simulated  disturbances  provided  or  to  be  supplied  by  pilot? 


Use  of  pilot-initiated  or  self-imposed  tasks,  disturbances,  or- short¬ 
term  maneuvers . 


Provision  for  familiarization: 


Measured  performance: 

Is  performance  to  be  measured? 
Pi lot- vehi cle  (output) Q 


Pilot  workload  ( input) PI 
Other  measurements? I  1 
Additional  provisions : _ 
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Are  additional  runs  allowed  for  evaluation? 


Explain: 


RATING  INFORMATION 

What  is  to  be  rated? 

□  Handling  qualities  □  Other _ _ _ 

□  Task  _ _ _ _ _ _ _ _ _ _ 

□  Flight  segment  (flight  phase  or  subphase) _ _ _ 

□  Composite  of  tasks  (specify  tasks  included  in  composite  rating) 


Q  Composite  of  subphases  (specify  subphases) _ 

What  is  provided? 

Test  vehicle:  (aircraft,  spacecraft,  simulator,  etc.) 


Aircraft  state: 

Normal  I  I  . Emergency  I  I  Identify  failure: 


Configuration: 


Gross  weight: _ '  _ _ _ 

Mass  distribution: _ _ _ _ 

Changes  in  aircraft  state: 

Configuration  changes  (during  task  or  designated  phase) 
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Transient  failures  (unanticipated  failures  to  be  introduced  ii^to  task 
for  pilot  reaction  to  and  correction  of  resulting  disturbance) 


Cockpit  interface : 

(Brief  description  of  important  items  noting  unusual  or  detracting  or 
limiting  characteristics  in  particular) 


Principal  controls: 
Primaiy  controls : 


Secondary  controls: 


Selectors : _ 

Cues  and  disturbances  provided: 


Motion : 


Visual ; 

Instruments:  Conventional d 
Full  panel'n 

(Brief  statement  of  instruments  or 
affect  the  evaluation) 


Novel  r~l 
Part  panel n 
characteristics  which  may 
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,  External  vision: 

External  visual  display  VFR  provided □  not  provided  □ 

IFR  only  □ 

(Brief  description  of  information  provided) 


Task(s)  : 

Control  task  description: 


Auxiliary  tasks?  Yes  n  NoD 
Aircraft  environment: 

Day  n  Night  Q 

Visibility  conditions: _ 

Turbulence  D  _ 

Wind  shear  [H  _ _ 

Crosswind  [U  _ 

Other  weather  conditions: _ 
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PILOT  EVALUATION  DATA 


Form  of  results 

Pilot  rating  □  Comments  Q  Questionnaire  Q  Oral  debriefing  Q 
Other  Q  _ _ _ _ _ _ _ _ 


Pilot  rating  scale: _ 

Principal  items  for  comment: 


Program  assessment: 

(Comments  and  recommendations  on  validity,  effectiveness,  etc. 
Confidence  factor  -  if  used) 


APPENDIX  B  -  Example  1 


BRIEFING  GUIDE  AND  RATING  INFORMATION  FOR  HANDLING  QUALITIES  EXPERIMENTS 


PROGRAM  DEFINITION 


Objectives 

General : 

Research  D  Development^  Acceptance  □  Other  □ 

Aircraft  role  Carmeroiat  transport _ 

Flight  segment  of  interest  Landing  approach _ _ 

Parameter  or  variable  of  primary  interest  Longitudinal  stability 


Mission  description: 

General  statement  of  the  required  operations  for  the  flight  segment 
of  interest  in  context  of  aircraft  role. 

1.  What  is  pilot -vehicle  combination  required  to  accomplish? 

Conduct  VFE  and  IFR  approaohes  and  ImdingSj  either  Day  or  Nighty 

with  an  aaauracy  ±S0  ft  from  dcsived  path  and  ±5  k  airspeed.  Ninety 

penoent  consistency  is  aiZowabZe  if  pilot  can  execute  wave  off  safety 

or  make  an  acceptdble  landing.  (5  ft/sec  at  touchdown^  within  50  ft 

of  centerline 3  with  side  velocity  less  than  5ft/sea) _ 


2.  What  are  the  conditions  under  which  these  required  operations  are 
to  be  carried  out  (i.e.,  aircraft  state,  environment,  and  cockpit 
interface)? 

Both  normal  oovi  en^rgency  state  with  pitch  SAS  faUsd,  Approach 

configuration i  Light-Moderate  turbulence^  Bight  or  Day^  0-20  k _ 

crosswind j  10  k  per  100  ft  windshear.  Cockpit  interface _ 

items  must  not  he  considered  limiting  or  to  introduce  significantly 
objectionable  characteristics  to  the  evaluation  pilots. _ 
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Scope 


Short  look  (guide  to  further  tests) Q  • 

Long  look  (definitive  as  possible)  ® 

Task  related  only  (minimize  pilot  extrapolation)  Q 

Mission  or  flight  segment  related  (pilot  extrapolation  encouraged) [1^ 
Nature  of  task  or  tasks  provided: 

Short .  term  D  Long  term  ® 

Critical  task(s)  included?  Q  Or  left  for  pilot  extrapolation?  ® 
Identify:  Consider  effect  of  highev  level  of  tupiulence  on  instrument 
apvroaah.  _ ^ ^ _ 


Are  simulated  disturbances  provided  or  to  be  supplied  by  pilot? 

Light  tvLpbuZenoe  and  wind  shear  provided.  _ 

Use  of  pilot-initiated  or  self-inqjosed  tasks,  disturbances,  or  short¬ 
term  maneuvers , 

Durirtg  farnttiarization  mens  only, _ _ _ 


Provision  for  familiarization:  five  vixns  at  beginning  of  each _ 

session. _ _ _ _ 

Measured  performance: 

Is  performance  to  be  measured?  Yes _ 

Pi lot- vehicle  (output)  ILS  tvadking  error _ 

Control  oolvann  ca^d  throttle  activity 
Pilot  workload  (input)  (reversals  and  work) _ _ 

Other  measurements?  Ei  Attitude  and  airspeed  error _ 

Additional  provisions :  All  but  familiarization  runs  used  only  to 
verify  learning  aurve. _ __________ 
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Are  additional  runs  allowed  for  evaluation?  Yes _ .  Explain: 

Pilot  may  make  nonpevformanoe  evalviation  mens  as  time  aVlows  after 
oamptetion  of  all  data  runs. _ 


RATING  INFORMATION 


What  is  to  be  rated? 

®  Handling  qualities  D  Other_ 
□  Task 


@  Flight  segment  (flight  phase  or  subphase)  Landing  cepproaoh 
D  Composite  of  tasks  (specify  tasks  included  in  composite  rating) 


Composite  of  subphases  (specify  subphasSs) 

is  provided? 

Test  vehicle:  (aircraft,  spacecraft,  simulator. 

Base  Simulator 

etc.)  S-16  Moving 

Aircraft  state: 

Normal  O  Emergency  Identify  failure: 

Pitch  SAS  failed. 

Configuration:  Approach  oonfiguration,  gear  ^ 

f  ,  flap  i  40° leading 

edge  extended^  wing  sweep  20^ ^  eta. 

Gross  weight:  Maximum  landing  weight 

Mass  distribution:  C,C,  at  aft. limit 

Changes  in  aircraft  state: 

Configuration  changes  (during  task  or  designated  phase) 


Flap  deflection  inoveased  from  mcmeuver  to  approach  upon 
inteveeptinq  glide  slope. _ 
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Transient  failures  (unanticipated  failures  to  be  introduced  into  task 
for  pilot  reaction  to  and  correction  of  resulting  disturbance) 

Pitch  SAS  failiope  to  he  introduaed  imexpeatedly  on  tmo  SAS  on  runs. 
Cockpit  interface: 

(Brief  description  of  inqportant  items  noting  unusual  or  detracting  or 
limiting  characteristics  in  particular) 


Principal  controls: 

Primary  controls:  Conventional  wheel  and  yoke  with  707  feel 
oharaoteristios;  single-throttle  control  for  all  engines. 


Secondary  controls:  Electric  trim  switch  on  wheel  for  -pitch  and 
voii^  ycoj  trim  -  not  provided;  spoiler  actuation  not  provided  in 
cockpit,  but  avaitdbte  on  voice  command. _ _ ■  -  • 

Selectors :  _ _ _ 

Cues  and  disturbances  provided:  Pitch  and  roZi  cues  with  turbuZence 

in  vertical  motion. _ 

Motion: 


Visual: 

Instruments:  Conventional  ®  Novel  D 

Full  panel  D  Part  panel 

(Brief  statement  of  instruments  or  characteristics  which  may 
affect  the  evaluation) 


External  vision: 


External  visual  display  VFR  provided  not  provided  C 

IFR  only  □ 

(Brief  description  of  information  provided) 

Closed  aiTouit  color  T.Y.  used  to  ^provide  vedHstio  visual  cu&s 
1500  ft  to  ground  aaaurate  14°  vertioal  ( dounvJccpd)  cutoff  angle 
pTovtdedf  but  lateral  view  restricted  to  40^  acngle.  Prominent 

horizon  provided.  Fairly  good  piaticpe  resolution  enables _ 

Z  ft /sec  touchdown  to  be  made  consistently  ^  but  hot  good  enough 

for  much  less, _ _ _ — 

Task(s) : 

Control  task  description;  a.  Vtsiuxl  approach  from  orosswind  leg^ 
70 00- ft  altitude  at  reoommndjed  approach  speed  to  land  at  5  000- ft 
elevation,  b.  IFR-ILS  approach^  aaptupe  and  traak  JJ5  from 
OM.  No  offset  drosswind  or  turbulence  2000  ft  RVR.  o.  Same  as 

h.j  except  with  20-k  crosswind,  light  turhulenae^  moderate _ 

windshear  at  100  ft  above  ground.  2000  ft  RVR _ 

Auxiliaiy  tasks?  Yes  □  No 
Aircraft  environment : 

Day  Night  ^ 

Visibility  conditions;  Test  varvdble _ 

Turbulence  Light  turbulence _ 

shear  1X1  10  h  per  100  ft  (SOO  ft  —  SO  ft  oltitudB) 

Crosswind  ^  ^0  h _ _ _ _ _ 

Other  weather  conditions:  None _ _ _ _ 
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PILOT  EVALUATION  DATA 


Form  of  results 

Pilot  rating  ®  Comments  □  Questionnaire  □  Oral  debriefing  ® 

Other  D  _ _ _ _ _ _ — - - - — - 

Pilot  rating  scale:  CoopeY-Barpev  - 

Principal  items  for  comment:  Airspeed  and  ftiafht-path  control.  Fitoh - 

response.  PiZoi  work-Zoad.  -  •  . — 

Program  assessment : 

(Comments  and  recommendations  on  validity,  effectiveness.,  etc. 

Confidence  factor  -  if  used) 

Validity  of  task.  Adeauaay  of  display.  -  - 


APPENDIX  B  -  Example  2 


BRIEFING  GUIDE  AND  RATING  INFORMATION  FOR  HANDLING  QUALITIES  EXPERIMENTS 


PROGRAM  DEFINITION 


Objectives 

General : 

Research  ®  Development  D  Acceptance  D  Other  D 

Aircraft  role  Long-rarufe  attack  borriber _ _ _ _ 

Flight  segment  of  interest  In-flight  re  fueling! _ 

Parameter  or  variable  of  primary  interest  LatevoZ  oontTol 
Mission  description: 

General  statement  of  the  required  operations  for  the  flight  segment 
of  interest  in  context  of  aircraft  role. 

1.  What  is  pilot-vehicle  combination  required  to  accomplish? 

By  visual  Tefevenoess  attain  and  maintain  -position  relative  to _ 

tanker  within  yvesented  hounds  for  yeriod  of  10  min. _ _ 


2.  What  are  the  conditions  under  which  these  required  operations  are 
to  be  carried  out  (i.e.,  aircraft  state,  environment,  and  cockpit 
interface) ? 

V  =  290  KIAS^  h  -  Z0»000  ft,  W  =  initial  and  refueled.  Dcoftimej, 
smooth  air  and  light  turhulenoe .  Straight  flight  ccnd  turn  entries. 
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Scope 


Short  look  (guide  to  further  tests)  D 
Long  look  (definitive  as  possible) 

Task  related  only  (ndnimize  pilot  extrapolation)  D 

Mission  or  flight  segment  related  (pilot  extrapolation  encouraged)  ® 

Nature  of  task  or  tasks  provided: 

Short  term  D  Long  term  ® 

Critical  task(s)  included?®  Or  left  for  pilot  extrapolation?  Q 

Identify :  Success  or  failvire  to  estcd>lish  and  maintain  position  with 

respect  to  tanker  with  VLciht  tuphulenoe  and  oQoassionaL  disturbance  . 

of  the  tanker. _ 

Are  simulated  disturbances  provided  or  to  be  supplied  by  pilot? 

Frovided  _ : _ 

Use  of  pi lot- initiated  or  self-imposed  tasks,  disturbances,  o-r  short¬ 
term  maneuvers. 

Only  during  famili.arization  or  subsequent  to  notification  of  test 

director. _ 

Provision  for  familiarization:  As  desired  hy  pilot. _ 


Measured  performance: 

Is  performance  to  be  measured?  Yes 
Pilot-vehicle  (output)  I®  _ 


Pilot  workload  (input)  D 
Other  measurements?  D 
Additional  provisions: _ 


47 


Are  additional  runs  allowed  for  evaluation?  Yes _ .  Explain: 


RATING  INFORMATION 

What  is  to  be  rated? 

®  Handling  qualities  Other  Adequacy  of  siwulation. _ 

D  Task^ _ _ _  ■ _ 

^  Fli|^t  segment  (flight  phase  or  subphase)  Special  phase  -  refuet-ing. 
D  Composite  of  tasks  (specify  tasks  included  in  composite  rating) 

D  Composite  of  subphases  (specify  subphases) _ 

What  is  provided? 

Test  vehicle:  (aircraft,  spacecraft,  simulator,  etc.)  Simulator _ 


Aircraft  state: 

Normal®  Emergency  Q  Identify  failure; 


Configuration :  Refueling 


Gross  weight:  500^000  and  400^000  Zb 


Mass  distribution:  AppropTiate 


Changes  in  aircraft  state: 

Configuration  chzmges  (during  task  or  designated  phase)  None 
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Transient  failures  (unanticipated  failures  to  be  introduced  into  task 
for  pilot  reaction  to  and  correction  of  resulting  disturbance) 

Icbj}  damper  _ _ _ _ _ _ _ 

Cockpit  interface: 

(Brief  description  of  inportant  items  noting  unusual  or  detracting  or 
limiting  characteristics  in  particular) 


Principal  controls : 

Primary  controls:  Control  stick  with  light  control  forces^  good 
centering  and  low  friction.  Conventional  rudder  pedals  hccoirtg 
light  forces  but  objectionably  poor  centering.  Requires  -pilot 

adaptation  ccnd  special  consider atian. _ _ _ _ 

Secondary  controls:  Trim  not  provided. _ _ _ 


Selectors :  Vot  provided  -  configuration  unchanged, _ 

Cues  and  disturbances  provided:_ _ _ _ 

Motion :  Pitchy  rolh  and  you  (±45°).  Accelerations  attenuated  ly 

limited  translation  (±5  ft  vertically,  ±10  ft  laterally; _ 

±10°  fore  and  aft).  _ _ _ 

Visual : 

Instruments:  Conventional  IS  Novel  Q 

Full  panel  D  Part  panel  D 

(Brief  statement  of  instruments  or  characteristics  which  may 
affect  the  evaluation) 

Considered  pznmarily  a  visual  tosh.  Only  primary  flight 
instruments  -  airspeed,  altitude,  and  rate  of  climb  prooid^d. 
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External  vision: 


External  visual  display  VFR  provided 


not  provided 
IFR  only  □ 


(Brief  description  of  information  provided) 

Closed  eirouit  T.  V. ,  enabling  visual  formation  flying  with 
stationary  model  of  tanker  aircraft.  _ _ 


Task(s): 

Control  task  description:  (1)  Close  on  tanker j  establish  contact ^ 

maintain  hookup  for  duration  of  refueling  operation  without _ 

thobulenee.  (2)  Same  as  (1) ^  except  for  simulated  light 

turhulenoe  and  ocoasional  abrupt  displaoement  of  tanker  to _ 

simulate,  disaonneot. _ _ _ 


Axixiliary  tasks?  Yes  D  No  ® 

Aircraft  environment: 

Day  IS  Night  O 

Visibility  conditions :  Unobstmioted  —  except  for  accurate 

windshield  and  visibility  representation. _ 

Turbulence  S  Light  _ _ _ _ 

Wind  shear  D  _ _ _ _ _ 

Crosswind  D  _ _ _ ^ _ — 

Other  weather  conditions  : _ _ _ _ 
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PILOT  EVALUATION  DATA 


Form  of  results 

Pilot  rating  IS  Comments  D  Questionnaire  S  Oral  debriefing  Q 
Other  D  Questionnaire  wilt  be  made  ava-ildble  after  completion  of  test. 

Pilot  rating  scale:  Cooper-Harper _ 

Principal  items  for  comment:  Pilot  workload.  Influence  of  control _ 

system  oharaoteristios 3  if  any,  _ _ 

Program  assessment: 

(Comments  and  recommendations  on  validity,  effectiveness,  etc. 

Confidence  factor  -  if  used) 

Realism  of  task.  Adequacy  of  simulated  motion. _ 
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